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Vorwort des Herausgebers 


Die Fahrzeugtechnik ist kontinuierlich Veränderungen unterworfen. Klima- 
wandel, die Verknappung einiger für Fahrzeugbau und -betrieb benötigter 
Rohstoffe, globaler Wettbewerb, gesellschaftlicher Wandel und das rapide 
Wachstum großer Städte erfordern neue Mobilitätslösungen, die vielfach ei- 
ne Neudefinition des Fahrzeugs erforderlich machen. Die Forderungen nach 
Steigerung der Energieeffizienz, Emissionsreduktion, erhöhter Fahr- und Ar- 
beitssicherheit, Benutzerfreundlichkeit und angemessenen Kosten sowie die 
Möglichkeiten der Digitalisierung und Vernetzung finden ihre Antworten nicht 
aus der singulären Verbesserung einzelner technischer Elemente, sondern be- 
nötigen Systemverständnis und eine domänenübergreifende Optimierung der 
Lösungen. 


Hierzu will die Karlsruher Schriftenreihe für Fahrzeugsystemtechnik einen 
Beitrag leisten. Für die Fahrzeuggattungen Pkw, Nfz, Mobile Arbeitsmaschi- 
nen und Bahnfahrzeuge werden Forschungsarbeiten vorgestellt, die Fahrzeug- 
systemtechnik auf vier Ebenen beleuchten: das Fahrzeug als komplexes, di- 
gitalisiertes mechatronisches System, die Mensch-Fahrzeug-Interaktion, das 
Fahrzeug in Verkehr und Infrastruktur sowie das Fahrzeug in Gesellschaft und 
Umwelt. 


Die Absicherung hoch automatisierter Fahrfunktionen ist allein im Fahrver- 
such nicht realisierbar. Die schier unendlich hohe Anzahl verschiedener Ver- 
kehrsszenarien würde bei einer reinen Teilnahme am Straßenverkehr sehr hohe 
Fahrstrecken erfordern, um jeder möglichen Konstellation zu begegnen. Dabei 
würde der Großteil der gefahrenen Strecke keine besonderen Herausforderun- 
gen für Sensorik, Algorithmik und Aktorik darstellen. Nur ein kleiner Teil der 
Fahrsituationen wäre zum Überprüfen eines angemessenen Fahrzeugverhaltens 
geeignet. Zur Steigerung der Effektivität und Effizienz von Absicherungsver- 
fahren ist die Identifikation sicherheitskritischer Verkehrssituationen und deren 
gezielten Verwendung zum Funktionstest erforderlich. 


Vorwort des Herausgebers 


Hier setzt die Arbeit von Herrn Elgharbawy an, in der er im Fahrversuch gewon- 
nene wissensbasierte Sammlungen risikobehafteter Szenarien durch automati- 
siert generierte Testfälle ergänzt, um so den vorgesehenen Betriebsbereich des 
Fahrzeugs möglichst vollständig abzudecken. Das Fahrzeugverhalten in diesen 
Testfällen bestimmt er in der Simulation oder auch ergänzt durch Hardware-in- 
the-Loop-Verfahren und leitet daraus die Wahrscheinlichkeit definierter Fehler- 
fälle sowie über eine Akzeptanzschwelle den Abbruch des Verifikationslaufs 
nach Erreichen der gewünschten Vorhersagegüte ab. 


Frank Gauterin 


Karlsruhe, im September 2022 


Kurzfassung 


Fahrerassistenzsysteme sowie automatisiertes Fahren leisten einen wesentli- 
chen Beitrag zur Verbesserung der Verkehrssicherheit von Kraftfahrzeugen, 
insbesondere von Nutzfahrzeugen. Mit der Weiterentwicklung des automa- 
tisierten Fahrens steigt hierbei die funktionale Leistungsfähigkeit, woraus 
Anforderungen an neue, gesamtheitliche Erprobungskonzepte entstehen. Um 
die Absicherung höherer Stufen von automatisierten Fahrfunktionen zu garan- 
tieren, sind neuartige Verifikations- und Validierungsmethoden erforderlich. 


Ziel dieser Arbeit ist es, durch die Aggregation von Testergebnissen aus wis- 
sensbasierten und datengetriebenen Testplattformen den Übergang von einer 
quantitativen Kilometerzahl zu einer qualitativen Testabdeckung zu ermögli- 
chen. Die adaptive Testabdeckung zielt somit auf einen Kompromiss zwischen 
Effizienz- und Effektivitätskriterien für die Absicherung von automatisier- 
ten Fahrfunktionen in der Produktentstehung von Nutzfahrzeugen ab. Diese 
Arbeit umfasst die Konzeption und Implementierung eines modularen Frame- 
works zur kundenorientierten Absicherung automatisierter Fahrfunktionen mit 
vertretbarem Aufwand. Ausgehend vom Konfliktmanagement für die Anfor- 
derungen der Teststrategie werden hochautomatisierte Testansätze entwickelt. 
Dementsprechend wird jeder Testansatz mit seinen jeweiligen Testzielen inte- 
griert, um die Basis eines kontextgesteuerten Testkonzepts zu realisieren. Die 
wesentlichen Beiträge dieser Arbeit befassen sich mit vier Schwerpunkten: 


e Zunächst wird ein Co-Simulationsansatz präsentiert, mit dem sich die 
Sensoreingänge in einem Hardware-in-the-Loop-Prüfstand mithilfe syn- 
thetischer Fahrszenarien simulieren und/ oder stimulieren lassen. Der 
vorgestellte Aufbau bietet einen phänomenologischen Modellierungsan- 
satz, um einen Kompromiss zwischen der Modellgranularität und dem 
Rechenaufwand der Echtzeitsimulation zu erreichen. Diese Methode 
wird für eine modulare Integration von Simulationskomponenten, wie 
Verkehrssimulation und Fahrdynamik, verwendet, um relevante Phäno- 
mene in kritischen Fahrszenarien zu modellieren. 
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Kurzfassung 


e Danach wird ein Messtechnik- und Datenanalysekonzept für die weltwei- 
te Absicherung von automatisierten Fahrfunktionen vorgestellt, welches 
eine Skalierbarkeit zur Aufzeichnung von Fahrzeugsensor- und/ oder 
Umfeldsensordaten von spezifischen Fahrereignissen einerseits und per- 
manenten Daten zur statistischen Absicherung und Softwareentwicklung 
andererseits erlaubt. Messdaten aus länderspezifischen Feldversuchen 
werden aufgezeichnet und zentral in einer Cloud-Datenbank gespeichert. 


e Anschließend wird ein ontologiebasierter Ansatz zur Integration einer 
komplementären Wissensquelle aus Feldbeobachtungen in ein Wissens- 
managementsystem beschrieben. Die Gruppierung von Aufzeichnungen 
wird mittels einer ereignisbasierten Zeitreihenanalyse mit hierarchischer 
Clusterbildung und normalisierter Kreuzkorrelation realisiert. Aus dem 
extrahierten Cluster und seinem Parameterraum lassen sich die Eintritts- 
wahrscheinlichkeit jedes logischen Szenarios und die Wahrscheinlich- 
keitsverteilungen der zugehörigen Parameter ableiten. Durch die Kor- 
relationsanalyse von synthetischen und naturalistischen Fahrszenarien 
wird die anforderungsbasierte Testabdeckung adaptiv und systematisch 
durch ausführbare Szenario-Spezifikationen erweitert. 


e Schließlich wird eine prospektive Risikobewertung als invertiertes Kon- 
fidenzniveau der messbaren Sicherheit mithilfe von Sensitivitäts- und 
Zuverlässigkeitsanalysen durchgeführt. Der Versagensbereich kann im 
Parameterraum identifiziert werden, um die Versagenswahrscheinlich- 
keit für jedes extrahierte logische Szenario durch verschiedene Stich- 
probenverfahren, wie beispielsweise die Monte-Carlo-Simulation und 
Adaptive-Importance-Sampling, vorherzusagen. Dabei führt die ge- 
schätzte Wahrscheinlichkeit einer Sicherheitsverletzung für jedes grup- 
pierte logische Szenario zu einer messbaren Sicherheitsvorhersage. 


Das vorgestellte Framework erlaubt es, die Lücke zwischen wissensbasierten 
und datengetriebenen Testplattformen zu schließen, um die Wissensbasis für 
die Abdeckung der Operational Design Domains konsequent zu erweitern. Zu- 
sammenfassend zeigen die Ergebnisse den Nutzen und die Herausforderungen 
des entwickelten Frameworks für messbare Sicherheit durch ein Vertrauens- 
maß der Risikobewertung. Dies ermöglicht eine kosteneffiziente Erweiterung 
der Validität der Testdomäne im gesamten Softwareentwicklungsprozess, um 
die erforderlichen Testabbruchkriterien zu erreichen. 
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Abstract 


Driver assistance systems and automated driving make a significant contribu- 
tion in improving the road safety for vehicles, particularly the commercial motor 
vehicles. With the further development of automated driving, the functional 
performance increases resulting in the need for new and comprehensive testing 
concepts. New verification and validation methods are therefore required to 
cope up with the testing of higher levels of the automated driving functions. 


This doctoral thesis aims to enable the transition from quantitative mileage to 
qualitative test coverage by aggregating the results of both knowledge-based 
and data-driven test platforms. The adaptive test coverage thus seeks to achieve 
a compromise between efficiency and effectiveness criteria of the assessment 
of automated driving functions in the product development of commercial 
motor vehicles. 


The systematic approach of this work includes the conception and implementa- 
tion of a modular framework for the customer-oriented testing of the automated 
driving functions with reasonable efforts. Based on conflict management for 
the requirements of the test strategy, highly automated test approaches are 
developed. Accordingly, each test approach is integrated with its respective 
test objectives to realize the basis of a context-driven test concept. The main 
contributions of this thesis are fourfold: 


e First, a co-simulation approach is presented which allows the perception 
sensor inputs to be simulated and/or stimulated in a Hardware-in-the- 
Loop test bench using synthetic driving scenarios. The presented setup 
offers a phenomenological modeling approach to achieve a compro- 
mise between the model granularity and the computational effort of the 
real-time simulation. This approach is used in a modular integration of 
simulation components such as traffic simulation and vehicle dynamics 
to model the relevant phenomena in critical driving scenarios. 


Abstract 


e Next, a data-logging and analysis approach for the worldwide validation 


of the automated driving functions is presented. This approach provides 
scalability for the acquisition of vehicle sensor and/or environment- 
perception-sensor data of specific driving events on one hand and per- 
manent data for statistical coverage and software development on the 
other hand. A cloud database centrally stores the acquired measurement 
data from the country-specific field tests. 


Subsequently, an ontology-based approach is described for integrating a 
complementary knowledge source from field observations into a knowl- 
edge management system. The clustering of recordings is realized by 
the means of an event-based time-series analysis with hierarchical clus- 
tering and normalized cross-correlation. The extracted clusters and their 
parameter space define the probability of occurrence of each logical 
scenario and the probability distributions of the associated parameters. 
Thereby, the correlation analysis of synthetic and naturalistic driving 
scenarios enlarges the requirements-based test coverage adaptively and 
systematically by executable scenario specifications. 


Eventually, a prospective risk assessment is carried out as an inverted 
confidence level of measurable safety using sensitivity and reliabili- 
ty analyses. The failure region is identified in the parameter space to 
predict the failure probability for each extracted logical scenario using 
sampling methods such as Monte-Carlo simulation and Adaptive Impor- 
tance Sampling. The estimated probability of a safety violation for each 
clustered logical scenario results in a measurable safety prediction. 


The presented framework allows a patching of the gap between knowledge- 
based and data-driven test platforms, thus consistently expanding the knowledge 
database of the Operational Design Domain coverage. In summary, the results 
show the benefits and challenges of the developed framework for measurable 
safety through a risk assessment confidence level. As a result, the validity 
of the test domain can be extended cost-effectively throughout the software 
development process to achieve meaningful test termination criteria. 
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1 Introduction 


The so-called Commercial Motor Vehicles (CMVs) are motor vehicles spe- 
cially designed to acquire economic value by transporting goods or individuals 
[FL*20]. Therefore, CMVs are highly specialized in fulfilling specific tasks 
and are primarily controlled by economic efficiency [Abe08]. In addition, the 
CMVs are characterized by a large number of series and models with tractors, 
semi-trailers or trailer combinations [T*17]. In most nations, the legislation 
regulates the concepts and functions of CMVs up to a specific vehicle system 
[ZMMFm13]. Moreover, the current challenges consist of improving road 
safety, addressing the scarcity of CMV drivers and making the profession 
of a CMV driver more attractive [Kirl5]. The vision of autonomous and 
accident-free driving thus, forces the further development of the automated 
driving technologies to be a current topic of high relevance in the transport 
sector [WRM*19]. Consequently, a broad range of political and economic 
stakeholders support this innovation in the expectation of reducing congestion, 
fuel consumption and accidents [MV 19]. 


1.1 Background 


In accordance with the World Health Organization (WHO), road accidents 
cause almost 1.35 million deaths and 20 to 50 million injuries every year 
[WHO20]. According to truck accident statistics, most accidents are caused 
by driver fatigue, lack of route information, as well as job pressure and 
aggressive driving [Kop22]. As reported by the United States Department 
of Transportation (USDOT), driver fatigue is a major cause of nearly 4,000 
fatalities in truck accidents on United States (U.S.) roads annually [Ass20]. 
Therefore, the United Nations General Assembly (UNGA) has launched a 
decade of road safety measures between 2011 and 2020 in order to reduce the 
risk of road accidents and injuries [fSIRTA22]. 


1 Introduction 


In 2015, the Federal STATIStical Office of Germany (DESTATIS) recorded 
accidents involving personal injury with the participation 
avy-duty truck in Germany. In spite of the accident variety 
with heavy-duty trucks, the statistics shows that rear collisions and unintended 
lane departures are the common types of CMV accidents with 68% of 32,500 
truck drivers [DES16]. Figure 1.1 depicts that the number of killed road users 
represent 2% of the total number of injuries and deaths with 787 fatalities. 
The three pie charts illustrate the percentage of types and causes of accidents 
involving the commercial trucks and the individuals involved on German roads 


a total of 29,480 
of at least one he 


in 2015. 
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passenger cars 
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Figure 1.1: Retrospective accident analysis of heavy-duty trucks on German roads according to 
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the DESTATIS report for 2015 [DES 16]. 


Types of accident consequences 


heavily injuried road users 


slightly injured road users 


1.1 Background 


On one hand, the truck accidents often have serious consequences such as injury 
and death, along with considerable financial impacts and environmental risks. 
Beyond the health and economic consequences, the truck accidents potential- 
ly cause significant losses on several levels such as reducing the efficiency 
of traffic flow and causing congestion [SWZ12]. On the other hand, freight 
traffic continues to increase globally and is the dominant means of transport. 
According to the traffic forecast for 2030, the volume of road freight transport 
in Germany will increase by 38% compared to the level of 2010 [M*20a]. 
Between 1992 and 2021, the number of truck accidents involving seriously 
injured road users has fallen by more than 59.0%. While the volume of truck 
traffic increased by 92.7% over the same period, the number of people who 
died in these accidents fell by more than 67.3%, as shown in figure 1.2. 


—@— fatalities —@— serious injuries —@— transport performance 


changes compared to 1992 levels 


Figure 1.2: Fatalities and seriously injured persons in truck accidents on German roads compared 
to truck transport performance between 1992 and 2021 [uEB22]. 


1 Introduction 


From a legal perspective, the road safety of commercial vehicles is an essential 
aspect of civil society. The European Commission (EC) has adopted the United 
Nations Regulation (UNR) nos. 130 and 131 to improve the safety of heavy- 
duty vehicles in the framework of the general safety regulation no. 661/2009. 
As a result, the installation of Autonomous Emergency Braking (AEB) and 
Lane Departure Warning (LDW) systems has become mandatory in Europe 
for all the new heavy-duty vehicles with a permissible gross weight of more 
than 3.5 tonnes. Only the off-road vehicles and trucks with more than three 
axles are exempt from these regulations [Elg12]. Moreover, the United Nations 
Economic Commission for Europe (UN/ECE) has introduced amendments to 
the Vienna Convention on Road Traffic (VCRT) in 2016 to explicitly allow the 
transfer of Dynamic Driving Tasks (DDTs) to Automated Driving Functions 
(ADFs) under the condition that these functions can be bypassed or switched 
off by the driver [MO17]. 


Definition 1.1 (AEB): An Advanced Driver Assistance System (ADAS) that 
can automatically determine the time required to perform the warning cascade 
and emergency braking in order to prevent the collision. Therefore, the AEB 
system represents an active safety function in the forward control of the CMV, 
in particular the control of the truck’s braking system. The AEB function 
includes a detection unit for measuring a distance between the Ego-vehicle and 
the relevant object ahead [ESS*19c]. 


Definition 1.2 (LDW): An ADAS that can warn the driver to prevent uninten- 
tional lane departure due to driver inattention or distraction. For this purpose, 
the LDW function utilizes the inputs from a front camera Electronic Control 
Unit (ECU) installed in the middle of the CMV behind the windshield. The 
camera ECU detects the lane markings and operates at speeds above 60 [km/h] 
to minimize false alarms related to construction sites and urban traffic [ESF19]. 


The VCRT of 1968 provides governments with a regulatory automobile 
framework for their national highways to ensure a high level of road safety 
for the contracting parties. For this reason, the Automatically Commanded 
Steering Function (ACSF) Category E within the UN/ECE Regulation no. 79 
corresponds to a function which is activated by the driver and which can con- 
tinuously identify the possibility of a maneuver (e.g. lane change) and perform 
these maneuvers over extended periods of time without driver confirmation 
[BHS17]. 


1.1 Background 


In addition, the USDOT published a framework in 2016 to support the safe 
development, testing and integration of automated vehicles [NHT17a]. More- 
over, in 2018, the Ministry of Industry and Information Technology (MIIT) 
of the Chinese government launched guidelines for the building of intelligent 
connected vehicles to expedite the development and review of standards for 
autonomous driving safety [WRM*19]. 


From a commercial perspective, heavy-duty and passenger vehicles differ in 
both their economic significance and the vehicle technology. The trucking 
business focuses on economic factors such as fuel consumption, truck utiliza- 
tion, driver demand and hours of service [LBS19]. A potential cut off in the 
cost therefore is a motivation for high driving automation in the long distance 
trucking. For fleet owners, financial benefits are of paramount importance in 
this context concerning the proposal of anew business case or insurance incen- 
tives [P*19a]. Thus, one of the key business cases is to achieve more service 
times for trucks and buses by supporting drivers or even substituting them with 
automated driving technologies [LSW18]. In this context, the niche strategy 
requires starting an application domain with a controllable environment in 
order to gradually expand the intended Operational Design Domain (ODD). 
According to the American Trucking Associations (ATA), the annual turnover 
rate of CMV fleet drivers is increasing, which indicates a strong demand for 
truck drivers [McN18]. In addition, it is expected that by 2026 the scarcity of 
truck drivers in the U.S. will reach over 174,000 drivers [CS15]. Accordingly, 
fleets respond to the scarcity of drivers by increasing the salaries to make the 
truck driver’s profession more attractive [McN19, BLS22]. 


From a technical perspective, there are several challenges to be overcome that 
affect the functionality of driving automation, such as significant variations in 
the state of vehicle loading, dimensions, weight, center of gravity position and 
braking performance of the heavy-duty truck. Despite the rapid advances in 
driving dynamics for enhanced safety, the requirements for heavy-duty vehicles 
differ considerably from those for passenger cars. In addition, CMV manufac- 
turers face special challenges of the relatively large numbers of variants with 
significantly lower production volumes and longer product life cycles. Table 1.1 
summarizes the different technical requirements for automation of passenger 
cars and CMVs. Therefore, long-haul CMVs are heavier, larger and less ma- 
neuverable than passenger cars [Har03]. 


1 Introduction 


CMV characteristics (e.g. dimension, low-speed transient off-tracking, braking 
distance, number of variants, etc.) therefore pose new challenges for ADFs. 
The off-tracking refers to a phenomenon in which the rear wheels track inside 
the path, traced by the front wheels, when a vehicle turns [ESS* 19b]. However, 
automated driving systems in the CMV sector are the most important business 
cases due to long-distance journeys, which sometimes reach more than 100,000 
kilometers per year on long and monotonous routes [Kir15]. 


Property Automated passenger car Automated CMV 
Niche market urban automation highway automation 
Mainstream market vehicle on demand services road freight transport services 
Potential user groups people with age-related or logistics and haulage 
medical constraints, teenagers companies 
and long-distance commuters 
360° surround detection feasible full rear view not feasible 
Detection range well standardized extended range due to 
longer braking distance 
Vehicle setup single body with multi-body with decoupled 
well-known structure cabin and unknown trailer 
structures 
Number of variants limited numerous 
Off-tracking negligible effect non-negligible effect within 
curve driving scenarios 
Tire type single mixture of single and twin 
Trailer hardly probable articulated trailers without 
environment sensing 
Supposed driver skills basic professional 
Driving scenarios complex selected routes on highway 


Table 1.1: Overview of the differences in automation requirements between passenger cars and 
CMVs. 


1.2 Automated Truck Driving 


Driver assisted trucks seek to improve the road safety either by warning the 
driver to avoid truck accidents or by directly taking control of the vehicle. 
With no claim to completeness compared with the relevant market, in 2006, 
Mercedes-Benz Trucks launched Active Brake Assist (ABA) 1 [TZ15] as the 
first emergency braking assistant in trucks that can prevent rear-end collisions. 
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1.2 Automated Truck Driving 


Consequently, since 2011 the Adaptive Cruise Control (ACC) has been sup- 
plemented with the Stop and Go function as an automatic distance control 
system. The ACC can identify the relevant preceding vehicles and calculate 
both the deceleration as well as the possible acceleration required to main- 
tain a safe distance. Due to the protection of Vulnerable Road Users (VRUs), 
such as pedestrians and cyclists, being a central aspect of safety development, 
ABA 4 and Side Guard Assist (SGA), both featuring pedestrian detection, 
were launched in 2016. The SGA can assist the truck driver in turning at low 
speeds when an object is laterally next to the heavy-duty truck or when the 
visibility is restricted by the vehicle length or due to adverse weather condi- 
tions. Furthermore, the SGA can detect moving and stationary objects in the 
warning zone on the co-driver’s side or in the turning curve and warn the driver 
visually and acoustically in critical driving situations [HSP19]. 


The ABA 4 can detect obstacles and VRUs by using multi-mode RAdio 
Detection And Ranging (RADAR) systems for short and long ranges and 
warn the driver of imminent collisions with pedestrians and simultaneously 
automatically initiate partial braking. The ABA 4 therefore allows the driver to 
avoid a collision by means of an emergency braking or a steering maneuver. In 
addition, the driver can warn pedestrians at risk by sounding the horn [Gol18]. 
In 2019, the fifth generation of the ABA has been launched into series produc- 
tion with improved pedestrian detection using a camera and RADAR system 
to monitor and detect the relevant preceding objects. The ABA 5 can warn the 
driver by a combination of optical, acoustic or haptic signals. Furthermore, the 
system can automatically determine the time required to perform the warning 
cascade and emergency braking in order to prevent the collision [Tru18]. 


In parallel, a partially automated driving system called Active Drive Assist 
(ADA) was launched in the market which can assist the driver in steering, ac- 
celerating and decelerating. In the case of lateral control, ADA can detect lane 
markings to keep the vehicle actively in lane via an electronically controlled 
steering system. As a result, the steering torque is designed to allow the driver 
to retain control of the vehicle at all times. The ADA also includes another 
function, called Lane Departure Protection (LDP), which can smoothly guide 
the vehicle back to the center of the lane in the event of an unintended crossing 
of lane markings following an acoustic warning. In the case of longitudinal 
control, ADA allows the truck to slow down on the approach to other vehicles 
in traffic and to accelerate again as the preceding traffic moves away. 


1 Introduction 


Accordingly, Daimler Trucks and Buses already met many important legal re- 
quirements in the area of road safety for CMVs many years prior to their entry 
into force [AG19, WRM*19, SB08]. For the self-driving trucks, the National 
Highway Traffic Safety Administration (NHTSA) has defined a set of safety 
design elements for the development and safeguarding of ADFs [NHT17b]. 
NHTSA encourages the automotive industry to provide these elements in the 
form of Voluntary Safety Self Assessment (VSSA) reports as guidelines for 
a safe deployment of ADFs on U.S. roadways. Potential users of truck auto- 
mation technology are logistics and freight forwarding companies as buyers 
of trucks [Fla16]. One of the motivations for a higher degree of automation in 
long-distance haulage is therefore a possible reduction in operating costs. The 
greatest potential benefit for users therefore arises when the driver is replaced 
by the technology. In fact, the prevailing shortage of truck drivers, rising cost 
pressure and low margins, as well as the growing need for efficient logistics 
processes, are driving the demand for automated trucks [MV 19]. 


By law, the companies that actively test their self-driving trucks on California’s 
public highways are required to disclose the number of kilometers driven in 
autonomous mode and the number of disengagements in which the test driver 
disengages the autonomous mode and takes immediate manual control of the 
self-driving truck. Accordingly, the vision of automated road transport in the 
logistics industry can serve to address the driver shortage, reduce costs and thus 
increase profit margins, and achieve greater process reliability by leveraging 
business cases, e.g. truck platooning and hub-to-hub autonomous trucking 
[MV 19]. 


Definition 1.3 (Truck Platooning): A linkage of two or more vehicles in con- 
voy, where the leading vehicle would be manned by a driver and the following 
vehicles would be electrically coupled trailers for certain parts of a journey. 
Truck platooning enables a business case for truck automation to increase road 
capability and reduce fuel consumption [ESFG18]. 


Definition 1.4 (Hub-to-Hub Autonomous Trucking): An application of the 
Transport-as-a-Service (TaaS) model that deploys trucks on predefined routes 
between depots or scheduled waypoints for shippers, carriers, logistics service 
providers and freight brokers [Sjo22]. 


1.3 Problem Statement 


With no claim to completeness compared with the relevant market, various 
autonomous truck companies have recently introduced concepts and prototypes 
for self-driving trucks and released their VSSA disclosures or disengagement 
reports according to the safety design elements outlined by the NHTSA, such 
as Daimler Trucks!, Waymo?, Volvo’, Tesla*, Einride>, Embark® [Rod19], 
TuSimple’ [TuS19], Kodiak Robotics® [Kod20] and Aurora? [Aur21]. 


1.3 Problem Statement 


The digital transformation has led to rapid and profound changes in the CMV 
industry. As a result, the future of road freight transport is changing in the 
era of connected and automated driving. From today’s perspective, global re- 
search and development activities are encouraging the Technology Readiness 
Level (TRL) of automated driving in the automotive industry. The digital 
transformation journey is thus fueled by the new driving force that is the data. 
The use of Field Operational Tests (FOTs) is indispensable to demonstrate the 
safety and reliability of these innovative technologies. Furthermore, the test 
CMVs equipped with data logging devices produce massive quantity of data 
on regular from public roads with the corresponding geographical distribution 
patterns. 


On one hand, their validation methods necessitate a higher level of bandwidth 
and storage of measurement recordings (e.g. automotive data communication 
buses, raw sensor detection lists, reference video streams, etc.). On the other 
hand, the use of these recorded data-sets is essential within software reprocess- 
ing and repeatable regression testing for a data-driven development of driving 
automation. 


https: //www.daimlertruck.com/innovation/autonomous-driving/our-path-to- 
autonomous-trucks.html 

2 https: //waymo.com/waymo-via/ 

3 https: //www.volvogroup.com/en-en/innovation/automation.html 

4 https: //www.tesla.com/semi 

5 https: //www.einride.tech/ 

6 https: //embarktrucks.com/ 

7 https: //www.tusimple.com/ 

8 https: //kodiak.ai/ 

9 https: //aurora.tech/ 


1 Introduction 


Apart from random hardware failures, logical and statistical failures can be 
distinguished as two categories of systematic software failures. The identifica- 
tion of logical errors demands meticulous analysis, convenient test equipment 
and proper functional decomposition. In contrast, the estimation of statisti- 
cal errors often requires a stochastic analysis of unintended reactions of the 
ADF in the recorded data-sets with the dynamic traffic situations of daily life. 
Consequently, the new highly complex technologies for automated driving 
need appropriate test strategies for a reliable and safe heavy-duty vehicle. 


The Automotive Systems Engineering (ASE) has established data-driven 
and knowledge-based test methods to ensure the required reliability and 
safety of ADFs. Data-driven approaches provide empirical evidence for the 
validation based on the Key Performance Indicators (KPIs). Moreover, vari- 
ous X(something)-in-the-Loop (XiL) simulations, ranging from microscopic 
to macroscopic traffic simulations and proving grounds, are used to enable 
efficient verification within the process of software product engineering. These 
knowledge-based approaches are often applied in a closed-loop setting to en- 
sure a requirement-based test coverage, defined by expert knowledge in the 
form of Natural Language (NL) statements. Consequently, implicit knowledge 
sources are extracted to transform them into explicit test specifications with 
the required granularity. Despite the systematic of the top-down approach, one 
of its major drawbacks is the assumption of knowledge completeness with 
restricted change of requirements during development. Besides, route-based 
assessment procedures offer a variety of situations with real traffic condi- 
tions [KRK*19]. Nevertheless, a significant reduction of the required driving 
mileages is unavoidable [JS* 19a]. 


According to the Society of Automotive Engineers (SAE) J3016, the status 
quo of ADFs extends to partially automated driving (SAE L2) [Int18]. In these 
systems, the driver retains control of the vehicle and remains obliged to mon- 
itor the functional intervention on a regular basis and, if necessary, to take 
over vehicle control. The decisive factors for the series development of these 
systems are the controllability of system interventions and the effectiveness 
in real traffic with minimal unintended reactions [WW16]. But a heavy-duty 
truck equipped with automated drive controlling can still be identified as a 
cyber-physical vehicle system in which the driving functions are designed to 
cope with dynamic traffic situations in an extremely safety-critical context. 
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1.3 Problem Statement 


Starting from SAE L3, a new type of vehicle guidance is implemented, in which 
the established test methods are neither sufficient nor appropriate [Win16b]. 
The main difference is that driver assistance may have unintended interventions 
that can be corrected by the driver. In the presence of functional inadequacies, 
the driver supervises the automated driving controller and performs the Object 
and Event Detection and Response (OEDR) sub-task [BDF* 14]. Accordingly, 
the human assisted driving functions are designed to be controllable at any 
time, but this can reduce their benefits [Weil3]. The controllability of system 
interventions and the effectiveness in the field with minimal undesired con- 
sequences are therefore decisive for the series development of these driving 
systems [W*18]. As a result, the ASE requires state-of-the-art evaluation pro- 
cedures to verify and validate these systems. The FOT is carried out to define 
thresholds for intervening systems based on the collected data. On one hand, 
trigger algorithms can be optimized to minimize the frequency and impact of 
falsely triggered interventions and, on the other hand, to maximize the number 
of legitimate responses. Nevertheless, driving automation requires the system 
to exploit the limits of DDTs and to master most environmental conditions 
controlled by a human driver [Sch15]. 


The International Organization for Standardization (ISO) 26262:2018 standard 
extends the functional safety regulations of Electrical and/or Electronic (E/E) 
systems for motorcycles and CMVs. However, the safety standard is limited to 
avoiding potentially safety-critical situations caused by systematic software and 
random hardware failures [AW 17]. Safety violations due to technological and 
technical deficiencies remain outside the scope of ISO 26262:2018 (e.g. insuf- 
ficient robustness, uncertainty issues with perception sensors, etc.) [BGH17]. 
Specifically, automated driving without driver monitoring can also lead to po- 
tentially safety-critical situations resulting from deficiencies in the estimation, 
interpretation and perception processes. Therefore, critical driving situations 
due to systematic software and random hardware failures can be handled within 
the ISO 26262:2018 standard. The ISO/ Publicly Available Specification (PAS) 
21448:2019 standard regulates the absence of unreasonable risk due to haz- 
ards resulting from functional insufficiencies of the intended functionality or 
by reasonably foreseeable misuse by persons [SH19b]. 


1 Introduction 


Definition 1.5 (ISO 26262): It specifies a development process for the func- 
tional safety of E/E systems in the automotive industry. It outlines a risk 
classification system and aims to reduce possible hazards caused by the mal- 
functioning behavior of E/E systems [Hil12]. 


Definition 1.6 (ISO/PAS 21448):It presents a guidance on the applicable 
design, verification and validation measures needed to demonstrate that there 
are no unreasonable risks arising from hazards due to performance limitations 
of the intended behavior [fS* 19b]. 


While there are no generally accepted test procedures at present that enable 
ADFs to be validated in affordable efforts, ongoing and completed German Fed- 
eral Ministry for Economic Affairs and Energy (BMWi) research projects (e.g. 
PEGASUS [WLFM19], AdaptIVe [Ete17], SafeMove [GA*18a, ADPZ19], 
KI-Absicherung [GGSB 19], SET Level4to5 [HTR*20] and VVM [ZRBE20]) 
show the relevance of research for new test methods. Accordingly, the main 
research questions are formulated as follows: 


RQ1: How can the knowledge-based and data-driven test platforms be 
combined in a complementary and collaborative manner? 


RQ2: How can the ADFs be effectively and efficiently tested? 


RQ3: How can the prospective risk be measured in order to achieve rea- 
sonable termination criteria for the testing of the ADFs? 


1.4 Research Objectives 


The statistical analysis of road accidents predicts the required mileage for 
levels of automation without driver involvement as a basis for the safety of 
new systems compared to their predecessors [JWKW 18]. These technologies 
face an unsolved challenge when it comes to proving safety during the de- 
velopment phase by means of FOTs. While the ADF uncertainties remain 
before automated driving is released for widespread use, it is essential to de- 
velop performance assessments for the safety confidence. Furthermore, highly 
automated test approaches are integrated to safeguard the ADFs. Black-box, 
grey-box and white-box tests are associated with their respective test objectives 
and form the basis for a context-driven test concept, as depicted in figure 1.3. 
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1.4 Research Objectives 
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Figure 1.3: Development and testing of ADFs using a trade-off between the efficiency and effec- 
tiveness criteria of a context-driven test concept [ESS*19c]. 
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1 Introduction 


The safeguarding process of the ADFs takes place in three stages in compliance 
with Automotive Software Process Improvement and Capability dEtermination 
(ASPICE) [VDA17, fS*19a]. In the first stage, there are two parts, namely 
white-box testing and grey-box testing according to the process of ASPICE 
Software Engineering Process Group (SWE).4 software unit verification. The 
white-box testing requires knowledge of internal software structures for func- 
tional and non-functional tests. The grey-box testing is the typical combination 
of the black-box testing and the white-box testing assessments that verify the 
functions of various components, systems or sub-systems to ensure maximized 
test coverage. 


The second stage is the database enrichment for open-loop and closed-loop 
tests according to the process of ASPICE SWE.5 software integration and 
integration test. The data of situation/scenario database goes through the open- 
loop test as well as the closed-loop test depending on the type of data ex- 
tracted [P*17a]. The ADF shall be adequately safe in the event of unintended 
reactions that could violate the safety goals. For this reason, the driver’s con- 
trollability or the Minimal Risk Condition (MRC) fallback shall confirm the 
probability of overcoming the system limits and failures. Also, the regression 
and progression testing ensure that the previously tested software remains at 
the same performance level, even if it is modified or combined with other 
software. Regression testing ensures that the maturity level of the software 
is retained while adding new logic and fixing issues. Meanwhile, progression 
testing tracks progress against specific ODD features. The progression tests 
serve as a benchmark against which developers can measure their progress. 
Based on the Minimum Viable Product (MVP) levels, the result of this stage 
decides whether more simulation tests are required or more test kilometers are 
needed. The MVP defines a version of a product with features to be usable by 
early customers who can then provide feedback for future product development. 


According to the process of ASPICE SWE.6 software qualification test, the 
development of ADFs is carried out using field based-observation. In the 
black-box testing, the data from various on-road tests, which are both func- 
tional as well as non-functional, is extracted. Depending on these test results, 
data-driven development provides the decision as to whether type approval is 
recommended or not [P*17b]. 


1.4 Research Objectives 


The presented framework uses the standard Quality Gates (QGs)!° for de- 
velopment of ADFs within an agile development process. Accordingly, the 
proposed concept aims to bridge the gap between knowledge-based and data- 
driven test approaches to enable continuous extensibility of experience in an 
adaptive test coverage manner. The final sign-off is carried out atthe end ofthe 
development process to ensure that the system meets the specified and intended 
requirements. This thesis describes a risk-based framework that provides a test 
process of sensing, perception, prediction and planning and motion control 
software modules. 


Definition 1.7 (Sensing): A software module indicates the ability of an ADF 
to receive adequate information from the vehicle’s internal and external envi- 
ronment through connected sensors [Ste16]. 


Definition 1.8 (Perception): A software module denotes the ability of an ADF 
to interpret information about its environment obtained through its sensors 
[Ber19]. 


Definition 1.9 (Planning): A software module defines the ability of an ADF 
to establish and navigate the route it will take on the way to its destination 
[SHE* 17]. 


Definition 1.10 (Motion Control): A software module characterizes the sys- 
tem’s ability to execute the driving functions necessary to carry out a contin- 
uously updated driving plan by delivering appropriate control inputs such as 
steering and braking [AKM17]. 


The modular framework represents a virtual testing for verifying ADFs on 
the ECU in the laboratory. The test bench offers an efficient compromise 
between the requirements of simulation realism and the real-time performance 
of the simulation environment. In this scheme, the real-world testing includes 
hierarchical clustering of recorded time-series signals to identify and assign 
the necessary test cases for different appropriate test environments. In addition, 
the structure employed utilizes a back-end database that is filled with catalogs 
of extracted driving scenarios from field-based observations. 


10 The QG indicates a special milestone that highlights the qualitative aspect of the deliverables 
at a defined point in a project. 
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1 Introduction 


Using an ontology-based method, a category of adequate and relevant logical 
scenarios for existing field tests is extracted. A semantic representation of 
concrete scenarios can be obtained using data mining techniques, and sys- 
tematically processed into executable requirements for ODD coverage. These 
new test cases then complement the existing test cases, which were developed 
from expert knowledge with NL based statements, in an adaptive test coverage 
manner. Moreover, the extracted scenarios and their parameter space define the 
probability of occurrence of each extracted logical scenario and the probabili- 
ty distributions of the associated parameters. The stochastic analysis methods 
employ surrogate models and sampling methods to calculate the probability of 
failure for each clustered logical scenario [Tar12]. The surrogate model refers 
to an interpretable model that is trained using input-output data to approximate 
the predictions of a black-box model [M*17]. The proposed procedure there- 
fore offers an optimized test strategy for extending the requirement-based test 
coverage in an adaptive manner. 


1.5 Structure of the Thesis 


The thesis is structured based on the flow required to support the above- 
mentioned research objectives. In this chapter, the thesis and its contributions 
are outlined. Chapter 2 discusses the safety assessment concepts of ADFs 
for long-distance CMVs. Furthermore, the ASE processes are reviewed. The 
fundamental concepts of software fault tolerance, which take deficiencies in en- 
vironmental perception and their management by multi-sensor data fusion and 
fail-operational architectures into account, are presented [Tun19]. In addition, 
the requirements for a measurable safety framework for different levels of 
automated driving are described. Chapter 3 presents an overview of the current 
state of scientific and technical knowledge on the test methods used in the 
work, which leads to all subsequent chapters. In addition, the challenges of test 
procedures for ADFs are discussed with respect to current research projects 
and standardization activities. 


Chapter 4 presents the integration of traffic simulation and truck dynamics into 
a co-simulation configuration based on trajectory-based control of road users. 
The sensor systems for the perception of the vehicle environment are simulated 
and evaluated. Thereafter, a round-trip latency in the distributed heterogeneous 
co-simulation environment is estimated. 
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1.5 Structure of the Thesis 


Chapter 5 illustrates various measurement methods for scenario mining. The 
data handling process of the in-vehicle data logging system and its main el- 
ements are explained with regard to event-based time-series analysis. Subse- 
quently, various methods of cluster analysis are discussed based on case studies 
of customer-oriented testing. 


Chapter 6 proposes the systematic process to complement virtual testing by 
extracting insights from field testing database using sensitivity analysis. In this 
thesis, the prototypical safety margin is the minimum Time To Collision (TTC). 
Thereby, the sensitivity analysis employs meta-model techniques, where the 
parameters extracted from cluster analysis are considered as input and the safety 
margins as output. In addition, the integration of ontology-based test scenario 
synthesis enables a systematic scenario enlargement. Chapter 7 concludes 
the framework implementation with the use of reliability analysis to estimate 
the probability of exceeding the safety margin. Accordingly, the reliability 
analysis explores the unsafe region in the parameter space to predict the failure 
probability for each logical scenario using sampling methods. Furthermore, 
some conclusions on the overall approach, limitations and recommendations 
for further possible developments are described in chapter 8 of this work. The 
mentioned flow is portrayed in figure 1.4. 
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Figure 1.4: Comprehensive overview of the thesis research process with three key parts (prelimi- 
naries, framework conception and implementation). 
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2 Safety Assurance in the Open 
Context 


The active and passive safety systems in the automotive industry differ in their 
assessment methods. A standard evaluation approach for the passive safety 
systems has been developed to assess the behavior in an appropriate number 
of crash test cases under certain critical conditions [Win16a]. Considering 
the active safety systems, there are many challenges to be overcome in safety 
Verification and Validation (V&V) under real traffic conditions, namely the 
diversity of scenarios and environmental conditions, system complexity, and 
functional deficiencies [HPT10]. Despite strong support from the industries 
and academia, questions are often raised about the business cases, ethical 
dilemmas, legal liability and safety regarding automated driving. The SAE 
J3016 automation levels are therefore expected to overlap and not be available 
on the market back to back. Due to a complex, uncertain and unpredictable 
traffic environment, the motion control and path planning algorithms have to 
deal with uncertainties in measurements and predictions [AB15]. Therefore, 
the automated driving relies on intelligent algorithms that receive input from 
a variety of environmental perception sensors to control real-time actions in a 
highly safety-critical context [ESW* 16]. 


2.1 Definition of Terms and Categories 


This section gives an overview of the basic terminologies and highlights the 
meaning of the basic terms used in this thesis. The essential phrases are 
categorized thematically to clarify the scope of the work. 


2 Safety Assurance in the Open Context 


2.1.1 Scene, Situation, Scenario and Test Case 


The interaction of ADFs, in conjunction with the traffic environment, increases 
the complexity of functional testing [M*18]. Thereby, the use case specifies 
the application, its desired behavior and its functional system boundaries. The 
use case description typically does not include a detailed list of all relevant 
scenarios for this use case. Instead, a rather abstract description of the deployed 
scenarios is used. 


Definition 2.1 (Scene): A snapshot of an environment with a certain state 
containing the scenery, dynamic elements, all actors and their relationships. 
The scenery includes all spatial stationary elements such as lane markings, 
traffic signs, obstacles and traffic lights [S* 18a]. 


Definition 2.2 (Situation): A selection of behavioral patterns for a specif- 
ic triggering event. The behavior reflects the interaction of the ADF with 
the environment. While the behavioral patterns deal with the allocation of 
responsibilities between the Ego-vehicle and its surroundings, the triggering 
event induces a traffic situation with certain conditions and subsequent system 
reactions [HSSB 14]. 


Definition 2.3 (Scenario): A historical development between several scenes 
in a chronological sequence of situations leading to a potentially hazardous 
consequence [BLO*17]. 


Definition 2.4 (Test Case): A specification of the inputs, execution conditions, 
testing procedure and expected results that aims to prove a particular property 
of a test object. Therefore, the test case contains a logical scenario with a set of 
parameters to determine whether a function operates according to its intended 
functionality [L* 18a]. 


2.1.2 Functional, Logical and Concrete Scenarios 


The layered structure of the automated driving system can be described as a 
cyber-physical system with a sensor system, an automated driving ECU and an 
actuator system, whereby the responsibility between the driver and the ADF is 
classified according to the SAE J3016 [Int18]. 
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Both the road and environment constraints consist of four layers of scenario 
description structured as follows: 


e Layer 1: road geometry (e.g. road curvature, lane marking, etc.), 
e Layer 2: static objects (e.g. speed limits, construction barriers, etc.), 
e Layer 3: dynamic objects (e.g. cars, pedestrians, trucks, etc.) and 


e Layer 4: weather conditions (e.g. fog, rain, snow, etc.). 


The scenarios can be classified into three levels of abstraction. One is functional, 
the second is logical and the third one is concrete. The functional scenarios 
specify the application, the desired behavior and the functional system bound- 
aries. The description of the functional scenarios doesn’t typically contain 
a detailed list of all relevant scenarios. Therefore, the functional scenarios 
illustrate the most abstract level of the scenario representations as high-level 
requirements with a textual or graphical description. The entities and their 
relationships are represented within the functional scenarios in NL statements. 
The logical scenarios represent precise specifications based on the parameter 
spaces in the stated space and contain a formal scenario description [BOS17]. 
The concrete scenarios define test cases and describe the concrete representa- 
tion of a logical scenario with predefined values in the stated space [Web19]. 


2.1.3 Structure-based, Open-loop and Closed-loop Testing 


The ADF contains software components that interact with an unstructured, 
public real-world environment to support and automate DDTs. Figure 2.1 
depicts the differences between structure-based, situation-based open-loop and 
scenario-based closed-loop testing in the time sequence. The particular func- 
tion provides the in-/output behavior or the reaction to sensor input variables 
in the automated driving system [SBWE19]. 


Definition 2.5 (Structure-based Testing): It is performed to determine the 
code coverage of software structure components such as specific software 
functions and code parts [Wil16]. 
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Definition 2.6 (Situation-based Open-loop Testing): It generates driving sit- 
uations from the required behavior and evaluates the function reaction without 
referring to future conditions [U*15]. 


Definition 2.7 (Scenario-based Closed-loop Testing): It is used to test the 
behavior in a closed loop within a traffic sequence of scenes associated with 
actions of the Ego-vehicle, events from the environment and goals for the ADF 
[FHW 16]. 
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Figure 2.1: Illustration of the logical relationships between structure-based, situation-based open- 
loop and scenario-based closed-loop testing [ESO* 19]. 


2.1.4 Operational Design Domain and Fallback 


The fallback method provides a MRC to reduce the risk of a collision if a 
particular route cannot be completed within the ODD. 


Definition 2.8 (ODD): According to SAE J3016, the ODD refers to the operat- 
ing conditions under which a given driving automation system or feature thereof 
is specifically designed to function, including environmental, geographical and 
time-of-day restrictions, and/or the requisite presence or absence of certain 
traffic or roadway characteristics [Cza18]. 
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Definition 2.9 (MRC): The MRC allows a driver or an ADF to switch to a 
low-risk operating condition using a DDT fallback. In case of a MRC, various 
fault tolerance strategies can be implemented to avoid a hazard to the system, 
so that the system continues with fail-safe, fail-degraded or fail-operational 
characteristic [CFHLO7]. 


The fail-safe behavior characterizes the transition of the system to a safe state 
despite the presence of hardware or software failures. The fail-degraded be- 
havior provides the safe-degraded property to run an intended degraded safe 
operation. In this context, degradation is defined as the reduced performance of 
the function which can still provide safe operation in the presence of hazardous 
events [Xi08]. The fail-operational behavior describes the ability to continue 
normal operation through redundancy of the components so that the loss of 
safety-related functions does not lead to a hazard [WRM* 19]. 


2.1.5 Verification, Validation and Accreditation 


In accordance with the Institute of Electrical and Electronics Engineers (IEEE) 
Std 610-1990, software verification denotes the process of evaluating software 
to determine whether the products of a given development phase satisfy the 
conditions imposed at the start of that phase. Meanwhilst, software validation 
describes the process of evaluating software during or at the end of the de- 
velopment process to determine whether it meets specified requirements. The 
ISO/ International Electrotechnical Commission (IEC)/IEEE 15288 describes 
a framework for characterizing the life cycle of systems and software engi- 
neering. The testing process requires three interrelated but distinct procedures, 
namely Verification, Validation and Accreditation (V VA), in order to develop 
a realization for a purpose to be fulfilled in a specific context [JS*19b]. 


Definition 2.10 (Functional Verification): According to the ISO/IEC/IEEE 
15288:2015, the functional verification refers to a procedure of confirmation 
by objective evidence that the specified requirements have been met [fS*15]. 


Definition 2.11 (Functional Validation): It concerns a confirmation procedure 
by objectively demonstrating that the requirements of a specific intended use 
or application have been fulfilled [P*17b]. 
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Definition 2.12 (Functional Accreditation):It relates to a confirmatory 
procedure by objective evidence that the function is acceptable for use in a 
specific purpose [fS*15]. 


These procedures collect and evaluate evidence to credibly determine whether 
an ADF can be safely used in a real-world ODD. Thereby, the deductive gap 
between required, specified and implemented behaviors refers to the presence 
of invalid hypotheses on different levels of abstraction that cause unintended 
functionality. Figure 2.2 represents three sets of S, M, and N in the set diagram, 
which create several overlapping areas when they intersect. The verification 
refers to a procedure that proves the correct implementation of each individual 
requirement to minimize the intersection area, K, so it is the set (MN SN N). 
The validation is a procedure that compares the results with observed empirical 
data to confirm the correctness of the requirements to minimize the intersection 
area, J, soitis the set (MN(S U N)). The accreditation is aimed at minimizing 
the areas A and D, so they are the sets (VM (M U S)) and (NN S N M), 
respectively. The adaptive functional testing aims to maximize the optimized 
behavior with the intersection area, C, so it is the set (SN M N N) and 
thus minimizes the deductive gap. If the deductive gaps are less than the 
reasonable risk, the continuous test process can be terminated according to the 
optimization goals by sign-off recommendations of ADFs. The areas 8 and £ 
are not critical where they have no safety-related impacts, so they are the sets 
(NA MNS) and (S N (M U N)), respectively [ESFG19a]. 
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Figure 2.2: Three-circles model of the VVA challenges due to the deductive gap between required, 
specified, and implemented behaviors [SBP*19]. 
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2.1.6 Top-Down, Bottom-Up and Middle-Out Approaches 


Software engineering is the systematic application of methods, principles 
and techniques to develop software-based systems in a consistent manner. 
Consequently, the top-down and bottom-up approaches refer to the design 
philosophies that are executed either from the top or the bottom for the 
requirements engineering [SS06]. In the software development process, the 
top-down approach requires a detailed design perspective of the system before 
the actual implementation can begin. In contrast, the bottom-up approach 
emphasizes developing the system by integrating the designed components. 
The middle-out approach is a mixture of the top-down and bottom-up ap- 
proaches and offers advantages over these when evaluating automated driving 
applications. The hardware and software development process is divided into 
several layers according to a V-shaped model (component, subsystem, system 
and vehicle), as shown in figure 2.3. 


Definition 2.13 (V-shaped model): It is a software development process model 
that combines the requirements and design on the left side with V&V on 
the right side. While the V-shaped model has established itself for series 
development in the automotive industry, it provides a procedure of abstraction 
and does not necessarily describe the test methods directly [Bac18]. 
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Figure 2.3: Adaption of the safety life cycle according to the ASPICE process model as a devel- 
opment process for ADFs [Sax08]. 
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The components consist of hardware as well as of software elements on the 
bottom layer. The subsystem therefore includes more than one component. 
Thus, the system can be divided into sub-systems with a hierarchical structure. 
The vehicle is comprised of one or more systems, whereby one system consists 
of at least one sensor, one processing unit and one actuator. A system imple- 
ments one or more functions, but a function can also be applied in several 
systems. 


2.1.7 Black-Box, White-Box and Grey-Box Testing 


The closed-loop testing uses XiL techniques for functional verification of 
software in synthetic simulation environments. For example, Hardware-in-the- 
Loop (HiL) platforms describe the functional verification of the embedded 
software integrated on the target ECU via technical interfaces. At the same 
time, open-loop recomputing performs logged data simulations and repeatable 
regression tests with many iterations to achieve functional improvements and 
parameter calibrations using recorded real-world data. The logged data simula- 
tions are based on sensor data-sets collected from physical test drives to select 
the most appropriate release version. The extensive data-sets are collected 
worldwide under realistic driving conditions with CMV fleets. The evaluation 
of algorithms requires an efficient search and interpretation of relevant traffic 
situations with the help of Root Cause Analysis (RCA) to modify the algorithm 
accordingly. 


The automated driving ECU shall be sufficiently safe in the event of unintend- 
ed reactions that could violate the safety goals. For this reason, the driver’s 
controllability or the fail-operational modes must confirm the probability of 
overcoming the system’s limits and failures. Moreover, the regression testing 
ensures that the previously tested software remains at the same performance 
level even if it is modified or combined with other software. A correlation 
between the various test coverage criteria intends to support the controlling 
whether to collect more kilometers or to carry out more simulations. To ensure 
that the system meets the specified requirements, a sign-off is carried out at the 
end of the development process [Lat08]. 


Definition 2.14 (Black-box Testing): It involves functional and non-functional 
tests with system description regardless of the component or system internal 
structures, such as on-road vehicle testing [Sax08]. 
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Definition 2.15 (White-box Testing): It requires knowledge of internal soft- 
ware structures for functional and non-functional tests. The code coverage then 
defines the parts of the software that are executed and those that are not, such 
as the Modified Condition/Decision Coverage (MC/DC) [Wil15]. 


Definition 2.16 (Grey-box Testing): It is a mixture of white-box and black-box 
assessments that verify the functional specifications of a component, subsys- 
tem, or system to ensure maximized test coverage [ESO* 19]. 


2.1.8 Software Development Process Models 


The software process is a set of activities for specifying, designing, implement- 
ing and testing software systems. The software process model is an abstract 
representation of a process that presents a description of a process from par- 
ticular perspective [P*19b]. The Software Development Life Cycle (SDLC) 
models specify the various stages of the process and the order in which they 
are carried out. Thereby, the SDLC models can essentially be divided into the 
following model types (Waterfall, V-shaped, Incremental, Spiral and Agile). 
The Waterfall model is a breakdown of project activities into sequential linear 
phases, where each phase depends on the deliverables of the previous one and 
corresponds to a specialization of tasks. In the V-shaped model, the relation- 
ships between each phase of the SDLC and its associated phase of testing are 
represented. The Incremental model is a method of software development in 
which the model is designed, implemented and tested incrementally until the 
product is finalized. The Spiral model is a risk-driven SDLC model in which 
the project is executed in loops. Each loop of the Spiral is referred to as a phase 
of the SDLC. In the Agile model, iterative development is used where iterations 
and continuous feedback are deployed to refine and deliver a software system. 


2.1.9 Requirements Specification Notations 


Requirements engineering refers to the process of eliciting, specifying and 
evaluating the desired behavior of a software-intensive system. The functional 
requirements form the backbone of a comprehensive technical understanding 
of the developed system. 
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Requirements as such, therefore, need to be unambiguous and understandable 
to allow an external testing organization to perform independent tests of the 
system. The NL-based requirements can be engineered in a straight-forward 
way without explicit knowledge of the syntax. Meanwhile, the model-based re- 
quirements facilitate the clarity of acomplex software product and enable asim- 
plified representation of the system with diagrams and axioms. The approaches 
of requirements management can essentially be divided into five notation types 
by using natural language and model-based notations as follows: 
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e Ad-hoc notation in NL is a free writing style that offers a high degree 


of flexibility in specifying requirements, but at the same time leads to 
ambiguity and a lack of expressiveness, completeness and consistency. 


Structured NL notation has less potential for ambiguity than an ad-hoc 
informal approach by requiring the use of NL in a structured format 
[C114]. 


Ontology-based NL notation uses a formal description language to 
describe an ontological knowledge base with a glossary of terms 
and relationships specified by a set of rules [STZ*11]. The ontology- 
based NL notation is a semantic human- and machine-understandable 
representation of knowledge terms and their inference rules. The ontolo- 
gies support the verification of the consistency and completeness of the 
requirements. 


Graphical modeling notation, based on finite state machines and sets 
using Unified Modeling Language (UML), represents use cases and/or 
sequence diagrams. The graphical model-based notation facilitates the 
clarity of a complex software product and enables a simplified repre- 
sentation of the system with diagrams or axioms. In spite of improved 
human readability, it may not be suitable for large systems where the 
graphical modeling notations are not easily maintainable. 


Mathematical notation provides a formal machine-readable format based 
on a set theory with Z notation specification [Bow01]. The Z notation 
is a formal specification language used for describing and modeling 
functional specifications. Therefore, the Z notation is a mixture of formal 
mathematical statements and informal text. The formal mathematical 
statements give a precise description of the system. 


2.1 Definition of Terms and Categories 


The informal text describes the meaning of the mathematical statements 
in NL to make the specification more readable. Large systems with a 
complex domain may not be easily specified in the Z notation, where 
formal specification is probably not readable and understandable to the 
client. 


Table 2.1 refers to a comparable evaluation where the scale ranges in require- 
ment notations from poor to optimal. The development of functional require- 
ments presents a joint process between the client and the contractor, in which 
the technical knowledge of the client and the software development competence 
of the contractor become accessible. Siegemund et al. [STZ*11] introduced a 
meta model for ontology-driven goal-oriented requirements engineering. In his 
dissertation, Siegemund [Sie14] derives the following quality criteria, which 
serve to analyze the requirements specification notations. 


Quality criteria Requirements specification notations 
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Table 2.1: Comparable evaluation of notation types in functional requirements engineering 
[ESO* 19]. 
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2.1.10 Software Process Assessments 


In safety-critical automotive software engineering, some quality process 
standards apply to organizational processes to achieve a high degree of soft- 
ware quality. ASPICE is a process reference model developed by organizations 
within the automotive industry to create a more automotive-focused reference 
model compared to ASPICE or Capability Maturity Model Integration (CMMI) 
[Bar08]. There are two dimensions in ASPICE: a process dimension and a ma- 
turity level dimension according to ISO/IEC 33001:2015. 


The ISO/IEC 33001 contains a set of technical standards for process evaluation 
to assess the achievement of process quality characteristics. The process dimen- 
sion includes various software development processes. The second dimension 
allows an averaged assessment of the maturity of a single process on a scale 
of O to 5, as illustrated in figure 2.4. The maturity level is determined by 
certified assessors who perform a comparison based on the processes defined 
in ASPICE [Win13]. The ASPICE enables the assessment and classification of 
their own process in relation to the state-of-the-art. Furthermore, the suppliers 
can be selected in a qualified manner and the improvement potential can be 
identified [BOS 19]. 
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Figure 2.4: ASPICE capability levels according to ISO/IEC 33001:2015 [Sch1 6a]. 
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2.1.11 Safety, Reliability and Availability 


The safety requirements are intended to ensure the absence of unreasonable 
risks at each stage of the development process of safety-critical automotive 
systems [IEC* 10]. Consequently, operational safety includes passive and active 
safety technologies to minimize the occurrence and consequences of traffic 
conditions. For this purpose, operational safety is divided into three main 
aspects: Functional safety, Safety Of the Intended Functionality (SOTIF) and 
behavioral safety. 


The functional safety describes the probability that a function does not go 
into an unsafe condition if an independent event may cause an accident. The 
functional safety focuses on system design to identify hazards using Hazard 
Analysis and Risk Assessment (HARA) and to alleviate the consequences of 
E/E malfunctions that may occur in the components of an automated driv- 
ing system. However, the goal of SOTIF is to validate the ADFs, including 
perception and decision making, in all the relevant environmental scenarios. 
Behavioral safety, meanwhile, focuses on the system design to behave safely 
in its environment to avoid hazards and reduce the risk of failures. 


Definition 2.17 (Reliability): It can be defined as the probability that a function 
fulfills its intended functionality in an ODD and over a certain period of time. 
The reliability requirements shall ensure that the system does not reach an 
unreasonable number of failures [Mul85]. Therefore, the reliability can be 
defined by the Mean Time Between Failures (MTBF) and the failure rate. 
While the MTBF describes the expected time between two failures of system 
components, the failure rate, in contrast, indicates the frequency with which a 
system component fails, in terms of failures per unit time [BA92]. 


Definition 2.18 (Availability): It is the probability that a system is available for 
operation at a certain point in time, i.e. the time period during which a system 
is actually in operation as a percentage of the total time that the system should 
be in operation. In parallel, robustness defines the degree to which a system or 
subsystem or component can operate correctly under invalid inputs or stressful 
environmental conditions [IEC*90]. 


2 Safety Assurance in the Open Context 


2.1.12 Fault, Error and Failure 


The error propagation model follows the subsequent steps: fault, error, failure, 
hazard and accident. The active errors are error classes that are caused by 
faults and cause failures, but the latent errors are error classes that are caused 
by faults and don’t cause failures [KMO5]. 


Definition 2.19 (Fault): An abnormal condition or defect that can cause an 
element or an item to fail. The faults occur as logical, Type I or Type II errors, 
which either lead to active errors or remain in the test object as latent errors 
[P*O1]. 


The Type I error, also called False Positive (FP) event, is an error type that 
occurs as a false alarm if the null hypothesis (Ho) is true, although the given 
condition doesn’t exist. While the Type II error, also called False Negative (FN) 
event, is an error type that occurs as missed detection if the null hypothesis 
(Ho) is false but the given condition erroneously fails to be recognized. 


Definition 2.20 (Error): A discrepancy from the intended design, which can 
lead to an unintended condition at the test object boundary. After that, a failure 
depicts the deviation from specified behavior due to undetected errors within 
the component, subsystem or system [SWB*20]. 


The occurrence of Type I and Type II errors summarizes the uncertainties 
resulting from the restriction of environmental perception and the variability 
of predictive situations. The Type I errors can provoke a new critical situation 
for endangering road safety. The Type II errors lead to a loss of the safety 
benefit concerning the system specification. However, each Type I or Type 
II detection error does not change the assessment of a situation as safe or 
hazardous. Therefore, the error criterion defines the formulation of the safety 
envelope that determines whether a situation is safe or hazardous. In addition, a 
safety-critical ghost or miss can be caused by measurement errors, e.g. a noisy 
distance measurement, or by detection errors, e.g. missing detection. For each 
detection or measurement error, there are two types of error for each signal 
value. The first describes the systematic error and the second the statistical 
error. The systematic error represents an error in the algorithm according to 
knowledge-based design. In contrast, the statistical error represents an error 
that occurs during operation due to environmental influences. 
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Safety-critical ghosts represent the rate of situations that are erroneously con- 
sidered hazardous, where ghosts can occur at any time. Safety-critical misses 
represent the situations that are erroneously considered safe, where safety- 
critical misses can only occur when a hazardous situation exists [SWB*20]. 


2.1.13 Microscopic and Macroscopic Risk Metrics 


The risk of the system can be estimated using a statistical approach to the 
frequency of accidents. Hazards at the system level are the physical situa- 
tions that may cause an accident. The accident is an unplanned or undesirable 
event that leads to an unrecoverable loss of service that typically brings loss 
and/or injury. Due to the long distance between two road traffic accidents, the 
macroscopic risk cannot be estimated without large field data. Therefore, an 
enormous mileage is required to collect enough accident data for a meaning- 
ful statistical analysis [JSW19]. Microscopic risk refers to the risk for single 
vehicle type or fleet of identical vehicles, e.g. a prospective risk of an ADF 
using Time To React (TTR) metrics. The macroscopic risk represents the av- 
erage risk in road traffic within a fleet of different vehicles, e.g. the occurrence 
rate of fatal accidents. If the investigated event is a critical scenario rather than 
an accident, the accident frequency rate increases and therefore less mileage 
is required for the equivalent statistical significance. Junietz [Jun19] applies 
microscopic risk metrics to evaluate critical scenarios and uses extreme value 
theory to extrapolate frequency of accidents using macroscopic risk metrics. 
The extrapolation is assumed with a hypothesis that is applied to ADFs to 
estimate the probability of accidents using macroscopic risk assessments. 


2.1.14 Sensitivity, Specificity and Precision 


The intended events contribute towards improving the safety of the ADFs. The 
True Positive (TP) event represents an intended reaction in which the system 
responds correctly to a critical situation. Similarly, the True Negative (TN) 
event refers to an intended reaction in which the system reacts correctly to a 
non-critical situation. The unintended events, in contrast, have different conse- 
quences: The FP event, also called Type I error, refers to an unintended reaction 
in which the system reacts incorrectly to a non-critical situation [Stel6]. As 
a result, the safety benefit is lost in this critical situation with regard to the 
system specifications. 
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The FN event, also called Type II error, means an unintended reaction that is 
not associated with a critical situation but can provoke a new critical situation 
[He114]. 


Definition 2.21 (Sensitivity): It evaluates how good the test is at detecting 
a critical situation and indicates the conditional probability that the system 
performs intended reactions to critical situations according to the system spec- 
ifications. 


The sensitivity, also called TP-rate, is calculated by dividing the number of TP 
events by the total number of TP and FN events, as defined in Equation 2.1. 
High sensitivity then refers to a low number of FN events [Ber16]. 


(2.1) 


sensitivity = P(positive reactions|critical situations) = TP4+FN 


Definition 2.22 (Specificity):It estimates how likely non-critical situations 
can be correctly ruled out and describes the conditional probability for the 
proportion of non-critical situations treated by the system. 


The specificity, also called TN-rate, is calculated by dividing the number of 
TN events by the total number of TN and FP events, as defined in Equation 
2.2. High specificity then refers to a low number of FP events. 


specificity = P(negative reactions|non-critical situations) = 


T 
—— (23) 
TN + FP 


The Receiver Operating Characteristic (ROC) curve is a graphical represen- 
tation of the relationship between sensitivity and specificity as performance 
measures for model selection. According to the German Insitute for Stan- 
dardization (DIN) ISO 5725-1:1997-11, precision is described by the standard 
deviation of the measured signal. In contrast, the accuracy is used to describe 
the signal bias to the ground truth value. The precision, also called Positive 
Predictive Value (PPV), is calculated by dividing the number of TP events by 
the total number of TP and FP events, as described in Equation 2.3. 


TP zj FP 
TP+FP TP + FP 


precision = (2.3) 
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2.1.15 Supervised, Unsupervised and Reinforcement 
Learning 


Machine learning approaches rely on computational statistics to make accurate 
predictions. In general, there are three categories of machine learning approach- 
es: supervised learning, unsupervised learning and reinforcement learning. 


Definition 2.23 (Supervised Learning): It focuses on approximating functions 
from a labeled data-set that can be used for classification and regression, i.e. 
Support Vector Machine (SVM), Naive Bayes, Logistic Regression, Decision 
Trees, Random Forest and K nearest neighbors. The supervised learning ap- 
proaches utilize inductive learning, in which a run-time algorithm uses the 
results of a learning process to perform algorithmic operations [K*19b]. 


Therefore, machine learning algorithms are evaluated to avoid systematic fail- 
ures during the training and validation process. Their specifications are there- 
fore not in the V-shaped model of a set of functional requirements for the 
system itself, but rather in a set of training data or a plan for capturing the 
set of training data [KW16]. If the machine learning process is not trained 
with a certain driving situation, the algorithm cannot recognize this situation 
and therefore the respective situation is considered as a Black Swan! situation. 
Therefore, the verification philosophy should consider Black Swan situations 
by robustness testing and run-time fault injection techniques. Thereby, the 
data collection process needs to reduce hazards such as unintended bias or 
distortion in the gathered data. The offline machine learning process starts 
with the acquisition of a training database using a collection of data from the 
on-road and proving ground tests. The data is then annotated to learn specific 
features (e.g. cars, pedestrians, trucks, etc.) and the scene labeling is used for 
the determination of parameters through training. The process of scene labeling 
is implemented by introducing artificial markers into a scene with the purpose 
of marking positions in a three-dimensional space. 


' Black Swan refers to a surprising event that is not observed by the system. Behind this term lies 
the story that in medieval Europe, people believed that there were no swans except white ones. 
After the discovery of the Black Swans in Australia, this surprising event broke the previous 
thinking [Tal10]. Therefore, the Black Swan is a metaphor that describes an event that comes 
as a surprise. 


35 


2 Safety Assurance in the Open Context 


Concerning self-verification, the training result is then verified with the train- 
ing data on the basis of acceptance criteria, such as acceptable Type Iand Type 
II error rates. The acceptance criteria refer to conditions that are defined to 
determine whether the test object can be delivered to the respective next test 
level. If the self-verification fails, the process can be restarted after collecting 
the additional data for the test abort criteria. The abort criteria relate to condi- 
tions under which the tests are terminated prematurely as continuation is not 
advisable. The self-assessment could be insufficient because it is difficult to 
ensure that the learning system has been trained for the essential features of 
training data rather than coincidental correlations. 


The coincidental correlation relates to a non-caused correlation of self- 
verification assessment criteria with training and validation data, which are 
consistent but are not caused by each other. Therefore, cross-verification is 
used to verify the learning process using a separately collected and annotated 
database with acceptable pass and fail criteria. The adaptive machine learning 
systems that change weights at run-time are relevant for the high and full auto- 
mated driving systems and go beyond the scope of the annotation and labeling 
process. 


Definition 2.24 (Unsupervised Learning): It focuses on discovering pattern 
in unlabeled data-sets with a minimum human supervision, i.e. hierarchical 
clustering, k-means, Density-Based Spatial Clustering of Applications with 
Noise (DBSCAN) and Principal Component Analysis (PCA) [KWB 18]. 


Definition 2.25 (Reinforcement Learning): It focuses on learning interaction 
from trial and error to take a decision in an environment based on maximizing 
the cumulative rewards, i.e. Markov Decision Process. 


The uncertainty quantification can provide information that is employed in ob- 
ject plausibility within sensor fusion algorithms [Min17]. There are two types 
of uncertainty that can be distinguished (Aleatoric and Epistemic uncertainties) 
[GB12]. Moreover, Type I errors (e.g. ghost objects, etc.), Type II errors (e.g. 
not detected objects, etc.) and misclassification issues can be tuned by the cov- 
erage of the training data-set. The performance of machine learning algorithms 
relies on the amount of training data-sets. The statistically relevant spread 
of operational situations can ensure adequate coverage during training for 
environmental perception tasks. 
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Definition 2.26 (Aleatoric Uncertainty): It comprises noises that are inher- 
ent to the observation (e.g. sensor or motion noises) [ODR*02]. Since the 
Aleatoric uncertainty refers to the self-noise of the observation, this uncertainty 
cannot be reduced by increasing the training data [R*20]. Therefore, Aleatoric 
uncertainty captures the observation noises that are inherent in the sensor 
systems, while the detection of a distant object may result in high Aleatoric 
uncertainty [VR15]. The Aleatoric uncertainty may lead to Type II errors and 
can be resolved by the complementarity of the environmental perception sensor 
systems [FRD 18]. 


Definition 2.27 (Epistemic Uncertainty): It has the effect that the system op- 
erates inconsistently for a given input class within a certain error range [VR15]. 
Therefore, the Epistemic uncertainty indicates that the model hypotheses may 
not adequately reflect reality for the intended functionality [HALZ19]. The 
Epistemic uncertainty or model uncertainty indicates how uncertain an object 
detector is to explain the object from the observed training data-set. The Epis- 
temic uncertainty may lead to Type I errors and can be addressed by more 
training data. For example, the detection of an abnormal object, different from 
the training data-set, may result in high Epistemic uncertainty [SKOK18]. 


2.1.16 Known, Unforeseeable and Unknowable Black 
Swans 


Safety reflects a result of the absence of unreasonable risks. Beyond the 
challenges of complying with accepted safety engineering procedures, a key 
challenge in the safety validation of automated driving is to behave appropri- 
ately in the presence of surprises. Thereby, it can be difficult or impossible for 
human reviewers to experience these events in advance [KW16]. The Black 
Swan events comprise three major categories of surprising critical scenarios 
in relation to present knowledge to verify ADFs, as follows: 


e The first type of Black Swan events comprises known critical scenarios 
that occur despite the fact that the probability of occurrence is judged to 
be negligible. Thus, acceptable risks shall not be determined exclusively 
by the probability assessment. The risk is a combination of the proba- 
bility of occurrence of harm and the severity of that harm. 
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In addition, the prudence and precautionary principles are fundamental 
elements of the risk management linked to such Black Swans. 


The second category of Black Swan events is unforeseen scenarios that 
are not covered by the corresponding risk assessment and management. 
Dealing with unforeseeable Black Swans demands improved risk assess- 
ment as well as knowledge discovery in order to identify unforeseeable 
surprises and include them in the relevant verification process to com- 
plement the present knowledge base. 


Unknowable Black Swan events represent the third category of uncertain- 
ty scenarios, which are completely unknown and only become known 
after the deployment of ADFs. The handling of unknowable critical 
scenarios demands that the ADF can automatically detect surprises in 
order to behave robustly and resiliently to uncertainties [WK15]. 


2.1.17 Safety Integrity Requirements for Automated Driving 


The SAE J3016 represents the levels of automation from 0 to 5, whereby SAE 
LO means no automation, SAE L1 applies to driver assistance, SAE L2 stands 
for partially automated driving, SAE L3 for conditional automated driving, 
SAE LA for highly automated driving and SAE L5 for fully automated driving 
[Int18]. The ADF is a combination of both hardware and software that can be 
equipped with one or more systems. ADFs with an automation level below SAE 
L3 enhance safety or assist the driver, but are not capable of controlling or oper- 
ating the truck without active physical control or monitoring of a human driver. 
ADFs with an automation level higher than SAE L2 are capable of performing 
the DDTs without active physical control or supervision by a human driver 
physically present in the truck cab. The safety integrity describes the reliability 
of a confidence indicator in conjunction with the functional requirements for 
the various automated driving levels. The safety integrity requirements for the 
respective automation levels are described, as follows: 


e The driver assistance and partially automated driving (SAE LO-L2) han- 
dle tasks of limited complexity autonomously in a precisely specified 
context. These functions perform limited tasks in a defined context and 
do not learn during operation. 
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Since the collaboration is a restricted task in a determined context, co- 
operation is therefore limited to the exchange of information on the 
system context. The safety integrity of ADAS and semi-automated driv- 
ing is aimed at ensuring the ISO 26262 HARA [SH19a] and ISO 21448 
SOTIF requirements [Hil12]. 


The highly automated driving (SAE L3) accomplishes a sequence of 
tasks, in which every single task is controllable, but the sequence and 
transitions between them are situation-dependent. While the system is 
not learning during operation, it optimizes its trajectories during the 
control process according to the defined objectives such as time or other 
resources. The co-operation with other systems is therefore limited to 
the exchange of information about the system context and the system 
itself. The safety integrity of highly automated driving shall ensure the 
feedback model from ISO 26262 HARA, SOTIF triggering events, and 
post-deployment driver experience to deal with the known Black Swan 
events in fail-operational mode using the cautionary and precautionary 
risk management principles [DH17]. 


The fully automated driving (SAE L4) is able to work together with 
other systems to perform their task. They negotiate their goals, plans 
and actions with other systems and adapt their behavior to the negoti- 
ated procedure. Since the system boundaries change dynamically due 
to the collaborative relationship, mechanisms for distributed planning 
and co-ordination of interpretations are required to ensure safe system 
functionality. Beyond the need to follow accepted safety engineering 
practices, the fully automated driving focuses on the safety requirements 
of ISO 26262 HARA, ISO 21448 SOTIF, ISO/DIS 34502 [fS*22a] and 
ODD coverage [fS*22b] to ensure reasonable behavior against unfore- 
seeable Black Swans using improved risk assessment and knowledge 
discovery principles [DH17]. 


Full automation (SAE L5) can expand the environmental perception, sit- 
uational awareness and actions with the ability of unsupervised learning 
and has some sort of fail-operational autonomy capability. The ANSI/UL 
4600 [UL22] safety case and continuous improvement feedback shall be 
required for the Autopoietic driving to safeguard a possible online ex- 
pansion. 


39 


2 Safety Assurance in the Open Context 


The Safety Performance Indicators (SPIs) and lifecycle feedback demon- 
strate the self-recognition and deal with unknowable Black Swans by 
defining the reasonable behavior. The SPIs are used to assess opera- 
tional safety performance through monitoring and measure validity of 
a safety case claim, e.g. violation of a safe clearance limit [KFFW 19]. 
Therefore, the system needs to be good enough to recognize surprises 
and ensure that the behavior remains relatively modest until the uncer- 
tainty is resolved. One of the safety skills is that the human driver has 
enough self-awareness to recognize an unclear driving situation and to 
minimize the risk until the uncertainty is eliminated. Therefore, contin- 
uous supervision and learning from field observations help in coping 
with rare and dangerous events. As a result, understanding the system’s 
own limits is essential when dealing with unknowable Black Swans. 
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0 no driving automation O O O O not available 
1 driver assistance © O O O limited 
2 partially driving automation © O O O limited 
3 conditional driving automation @ © © O limited 
4 high driving automation @ © © © limited 
5 full driving automation © © © ®© unlimited 


O : human driver 
© : human driver and/or system 
©: system 


Table 2.2: ODD scope and role-sharing between the human driver and the system at each automa- 
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tion level, adapted from the SAE J3016 automation levels [Int18], [ESO* 19], [DH17]. 


2.2 Uncertainties in the Environment Perception 


The ODD scope and DDT fallback requirements for each level of automa- 
tion, as illustrated in table 2.2 [Smil7]. The DDTs comprise all real-time 
operational functions required for the operation of a vehicle in road traffic, 
including environment perception, longitudinal and lateral motion control and 
maneuver planning. 


2.2 Uncertainties in the Environment Perception 


Since automated driving depends on the perception of the vehicle’s environ- 
ment, safety violations can be caused by system restrictions due to physical or 
technical limitations of the intended functionality [Cho16]. Table 2.3 refers to 
a comparable evaluation with a scale from poor to optimal using the following 
twelve sensor capability criteria applied to the environment perception of 
CMVs. The perception and map-based prediction sensors have different mea- 
surement principles, which are generally divided into camera, RADAR, Light 
Detection And Ranging (LiDAR) and electronic Horizon (eHorizon) sensors. 


Steinmeyer et al. [S*18b] introduced a method for improved data fusion in an 
environment detection. In his dissertation, Steinmeyer [Ste14] evaluates the 
environment perception sensors based on different sensor capability criteria, 
e.g. maximum longitudinal range, lateral Field of View (FOV), etc. The object 
recognition and classification tasks are performed by machine learning tech- 
niques to extract relevant characteristics in an unstructured operational context. 
While machine learning paradigms offer a promising perception performance, 
high levels of Type I and Type II error rates can decisively influence the 
functional safety of the overall system. Therefore, the performance evaluation of 
the environment perception should be defined to ensure a sufficiently safe level 
of residual risk associated with functional deficiencies in machine learning 
algorithms. For this, various sensors must be verified not only concerning 
their failure rates, but also about the possible causes of technical shortcomings 
in machine learning. Consequently, the quantitative evaluation of perception 
sensors and algorithms should consist of Type I and Type I error rates, in which 
some assumptions about the system context are implied. Thus, the robustness 
in real traffic can be achieved by the creative fusion of sensor data as well as 
appropriate system design. 
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Sensor capability criteria 


Environment perception sensors 


Maximum longitudinal range 


Lateral field of view 


Longitudinal range accuracy 


Lateral range accuracy 


Relative object speed estimation 


Moving object dimension 


Moving object classification 


Adverse weather conditions 


Behavior at darkness 


Sensor installation flexibility 


Sensor cost requirements 


Road classification 


© 0 O| O| O| © O @| © © ©) Camera-based perception 


©! @| @| @| @| ©0|0|® O ®© © @| RADAR-based perception 
© O ©| @| C| O| ©} © © © © © LiDAR-based perception 


@| 0 @| @| @| CO] O} CO] CO} CO} @| @| eHorizon perception 


@: optimal @: fairly optimal 
@: natural ©: fairly poor 
O : poor 


Table 2.3: Comparable evaluation of environmental perception and situation prediction sensors 


[Ste14]. 


2.2.1 Camera-based Perception 


The camera sensors measure the incident light using an optical system and cap- 
ture visible cues similar to human perception. The exposure control regulates 
the exposure with constant contrast regardless of changes in brightness and 
direction. However, no depth or speed information can be measured directly, 
whereby the three-dimensional world is projected onto the two-dimensional 
image. The recognized object features are mapped to a vector representing an 


object hypothesis in the state space of the used classifier. 
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2.2 Uncertainties in the Environment Perception 


The stereo vision sensors consist of two monocular cameras with a certain 
distance between each other, typically known as base width, and measure an 
environment detail from different perspectives. The measurements are carried 
out synchronously, whereby a depth estimation is generated in the two images 
by comparing the displacement (disparity) of individual pixels or patterns. 
Besides, the distance accuracy is limited by the resolution, especially at long 
distances. Today’s automotive cameras play a vital role in environment per- 
ception due to the high information density present in images. However, its 
drawbacks are, similar to automotive LiDAR, weather sensitivity and limited 
detection range. The following causes can lead to Type I errors when camera- 
based perception has low specificity: 


e False object hypotheses due to ghost objects or bright lights. 
e Ambiguities in the disparity calculation through repetitive patterns. 


e Underexposed backgrounds due to color distortions. 
Type II errors due to low sensitivity can have the following causes: 


e Poorly illuminated objects. 

e No pattern matching within the training data-set. 

e Objects with low disparity due to homogeneous surfaces. 
e Objects with low height due to no separation by layer. 

e Overexposed backgrounds due to direct sunlight. 


e Overexposed backgrounds due to adverse weather conditions (e.g. fog, 
snow, rain, etc.). 


2.2.2 RADAR-based Perception 


Range Sensors such as LiDAR and RADAR calculate the distance, angle and 
signal power to detect targets in a particular region of interest. Automotive 
RADAR sensors observe the position and velocity of moving objects as well as 
stationary roadside objects with precise range information and high resistance 
to adverse weather conditions. 
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However, RADAR detection is afflicted with a limited angular resolution in the 
case of stationary or longitudinally moving pedestrians. Automotive RADAR 
sensors transmit and receive radio waves to determine the velocity, range 
and angle of objects in the 76-81 GHz band. Its strengths derive from an 
extended longitudinal range, an optimal accuracy of the range rate with weath- 
er independence. RADAR sensors, however, can poorly resolve closely spaced 
objects over long distances. The Type I errors due to low specificity can have 
the following causes: 


Extended metallic objects that can be driven over (e.g. road sign gantries, 
road bridges and overpasses, tunnel fans, corrugated sheets, etc.). 


Extended metallic objects that can be driven under (e.g. guard rails, 
movable manhole covers, beverage cans, etc.). 


Ambiguity effects in object classification due to insufficient resolution 
in frequency tuning (e.g. through alley situations). 


e Ambiguities with an extended activated field of view due to the higher 
deceleration time of commercial vehicles. 


Specular reflections and noise detections during Constant False Alarm 
Rate (CFAR) detection. 


The following causes can lead to Type II errors when RADAR-based perception 
has low sensitivity: 


e Objects with low Radar Cross Section (RCS) values. 


e Aging affected radome behind the bumper with different damping char- 
acteristics. The radome is a plastic housing to shield the RADAR antenna 
from weather influences. 


2.2.3 LiDAR-based Perception 


Automotive LiDAR sensors are based on an optical measurement principle 
to locate and measure the distance of objects in space. The LiDAR sensors 
typically use the time of flight principle for distance measurement where a 
laser pulse is emitted and the elapsed time is measured until the reflected 
signal is received again. 
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The time delay between transmission and reception is directly proportional to 
the distance due to the proportionality between the time of flight and distance. 
The possible causes of Type I errors for LiDAR-based perception are poorly 
illuminated objects and high road inclination. At the same time, those for the 
Type II errors are light-absorbing objects, planer surface objects and adverse 
weather conditions. 


2.2.4 eHorizon-based Perception 


There are several standardized solutions related to High Definition (HD) maps 
to meet the challenges of further developing the driver assistance to a higher 
level of automated driving [Sas17]. The SENSOR Interface Specification 
(SENSORIS) protocol defines a standard to collect data from the sensors 
of environment perception to the map provider. The Ego-vehicle receives the 
necessary traffic information from the map provider via the Transport Protocol 
Experts Group (TPEG) protocol. The cloud-based exchange maintains incre- 
mental updates for digital maps in navigation and infotainment systems via the 
Navigation Data Standard (NDS) protocol, as demonstrated in figure 2.5. 


The ADAS Interface Specification (ADASIS) protocol defines a standardized 
data model and interface that ensures regular interaction between the ADFs 
that generate or use the eHorizon. In the example of a Traffic Sign Recognition 
(TSR) function, the fusion of information sources, i.e. data from the eHorizon, 
image processing data and vehicle data, makes it possible to recognize the 
maximum admissible speed more reliably and conveniently [J*11]. 


The hybrid data representation of detailed digital maps and physical automo- 
tive sensors provide an extended view of the Ego-vehicle environment and 
thereby facilitate improved inferences and more competent decision-making 
[MPS*15]. Incidentally, digital maps are designed to be updated from time to 
time. Janssen et al. used the Dempster-Shafer fusion technique to merge digital 
map road signs with road signs detected by a video system [JNO4]. Nienhuser et 
al. also employed the same fusion technique to give the corresponding priority 
to each data source [N*09]. The decision level fusion functions to validate 
inputs of higher levels and increase the robustness of the overall system in 
complex scenarios. 
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Figure 2.5: Proposed architecture by Open AutoDrive Forum in the context of HD 3D maps using 
the example of TSR function [Sas17]. 


Since today’s standard vehicle sensor measurements contain relatively limited 
information content, the eHorizon data serves as a predictive sensor to antici- 
pate the driving path. The eHorizon sensors employ the digital map data and 
Global Positioning System (GPS) sensors to predict the driving route. The GPS 
sensor determines the vehicle position in world co-ordinates [Bra13]. The map 
matching transforms this place into map coordinates and assigns it to a specific 
road on the map [ZBIG12]. Subsequently, the Most Probable Path (MPP) ex- 
tracts and processes the relevant map characteristics along the most likely future 
route using prediction algorithms [Kv07]. The eHorizon data includes vehicle 
position data and road segment attributes such as road geometry, road class, 
number of lanes and speed limits [JS14]. On the transmitter side, eHorizon 
data is extracted by an eHorizon provider along the MPP and delivered to the 
vehicle bus system. 
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On the receiver side, an eHorizon re-constructor decodes the vehicle bus mes- 
sages and transfers the data to the fusion module [DS11]. The discrepancies 
between GPS position data and matching maps are the probable causes of the 
Type I errors, while the usage of non-updated map data in memory due to the 
changes in traffic conditions are the possible causes of the Type II errors. 


2.3 Data Fusion of Environment-Perception 
Sensors 


Despite the recent rapid growth of human assisted driving systems to consoli- 
date road safety, they still have various challenges when coping with dynamic 
traffic situations of daily life. Moreover, the issue of protecting VRUs is that 
their movement is mostly unpredictable [CAR15]. The VRUs are defined as 
non-motorized road users who use a road including sidewalk and other adja- 
cent spaces, such as pedestrians, cyclists and persons with disabilities or limited 
mobility and orientation. Hence, there is an urgent need to locate the spatial and 
temporal coverage problems using an adequate environment model for situation 
evaluation, which is based on a synergistic approach between the existing 
sensors located on the body surface present in the truck’s sensor network 
[OPLS11]. Thereby, the collaborative multi-sensor data fusion becomes in- 
creasingly critical to provide a better understanding of the monitored area. As 
a result, fusion algorithms analyze and interpret the traffic situation to provide 
a clear situation analysis and derive suitable control measures. 


There are various ways to categorize the structures and methods of data fusion 
approaches. A distinction can be made between the architecture, the abstrac- 
tion level of the input data and the sensor integration. Also, it is possible to 
distinguish between implicit and explicit fusion approaches as well as grid- 
based and parametric approaches. Basically, the fusion system can be divided 
into three primary fusion types: complementary to supplement incomplete 
sensor data, redundant to reduce erroneous measurements and cooperative to 
improve quality of environment model [Ott13]. The recognition of the vehicle 
environment for a comprehensive understanding of the scene is one of the 
most important elements for the realization of an ADF. Therefore, the use of 
different sensor technologies like camera, RADAR, LiDAR and eHorizon with 
different detection capabilities is necessary to provide both complementary 
and redundant information for the data fusion unit. 
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The fusion unit analyzes and evaluates the different sensor signals and generates 
a dynamic surround model with a good scene understanding. Although current 
ADFs (SAE L0-L2) can only use objects to generate a simple surround model, 
future stages of automated driving (SAE L3-L5) will need to combine not only 
the objects but also the sensor-specific features and characteristics of these 
objects. Hence, data fusion can often occur at three logical interface levels 
(detection, feature and object level). First, detection level fusion performs 
fused detections in the earlier steps of the processing chain to validate lower 
level inputs and increase the robustness of the overall system in complex 
scenarios. However, detection level fusion combines recognized detections 
without classification, model-based filtering and tracking. The origin of the 
detection co-ordinate system is the position of the sensor mounting in the ve- 
hicle co-ordinate system. Next, the feature level fusion provides recognized 
features that are merged based on classified sensor-specific attributes in the 
vehicle co-ordinate system without model-based filtering or tracking. Finally, 
the recognized object entities are tracked and filtered over time. Model-based 
algorithms classify the object entities (e.g. potentially moving objects, static 
objects and road markings) and evaluate them with existence probability values. 
Therefore, each recognized object entity has a unique object identification 
value that does not change over time. The object identification value can only 
be reused when the object is no longer visible. 


The ISO/ Draft International Standard (DIS) 23150 is an automotive standard 
that addresses the logical interface of data communication between perception 
sensors and data fusion unit for ADFs [fS*21]. The data fusion unit intends to 
utilize the overlapping FOVs of the employed environment perception sensors 
and produces accurate information with a low rate of false-alarm detections 
[S*09]. Duraisamy and Schwarz have given a survey of the track-to-track data 
association methods using a decentralized sensor fusion architecture in order 
to achieve an optimal fused result [DS15]. Despite its significant contribu- 
tion to situation awareness, one of its primary drawbacks is the increased 
software complexity caused by multiple interconnecting sensors [DFS* 10]. 
The abstraction level at which the data fusion takes place is a trade-off between 
information context and complexity. While environmental perception sensors 
are still afflicted with adverse weather conditions, such as snowing or raining, 
HD maps can cope well with implicit traffic information such as speed limit 
changes through the highway-to-city transition [W* 14]. 
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Intelligent transportation systems use various sensor technologies to perceive 
the vehicle environment. Automotive RADAR sensors are commonly employed 
for object and pedestrian detection with precise range information and high re- 
sistance to adverse weather conditions. However, RADAR detection is afflicted 
with limited angular resolution in the case of stationary or longitudinally mov- 
ing pedestrians. These days, the automotive cameras also play a decisive role 
in environment perception due to the high information density in images. 


Figure 2.6 illustrates the functional components of an automated driving sys- 
tem relying on object-level data fusion. The tracked object properties with 
corresponding object list are shown in table 2.4. The layered structure of the 
multi-sensor fusion system can be described as a cyber-physical vehicle sys- 


tem with a sequence of sensor nodes N,, where k € [1, N,]. Each sensor node 
(k) 


S‘“ maps a set of sensor detected objects P(X = Ip! 
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into the object-level data fusion algorithm, where the indices i and j enumerate 
the objects in the respective set. The detections of object and lane boundaries 
refer to unique IDs. The existence probability represents the detection quality 
and certainty of an object. A higher value indicates an object which exists with 
high probability and a lower value indicates a potential false alarm object. 
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Figure 2.6: Functional components of an automated driving system including an object-level data 
fusion module [ESW* 16]. 
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Object property Object list Q% 

Object ID unique ID of the detected object or lane marking 
Detection history time stamp at the global synchronized time 
Relative distance longitudinal and lateral 

Relative velocity longitudinal and lateral 

Relative acceleration longitudinal and lateral 

Object deviation longitudinal and lateral 


Relevance object selection lane assignment (e.g. left, ego and right lane) 


Bounding box dimension width and height 

Movement status object state (e.g. stopped, moving, stationary, etc.) 
Classification probability object type (e.g. truck, pedestrian, etc.) 

Existence probability object detection quality and certainty 

Lane boundary type and color of the classified lane marking 

Criticality level confidence of the classified object or lane boundary type 
Road specific data route information (e.g. construction area, exit lane, etc.) 
Additional data sensor specific data (e.g. angular velocity, etc.) 


Table 2.4: Data structure of object level fusion for environment perception sensors [ESFG19b]. 


In general, the existence probability depends on two factors: quality and confi- 
dence of the measurement used to generate the object and the object’s observ- 
ability over time. The measurement quality model is usually closely associated 
with a sensor’s specific characteristics, for example RCS or a camera classifier. 
Equation 2.4 represents the signal flow chain of sensor S(®, where A, 
B“ and C'® represent characteristics of measuring principle, processing and 
observation, respectively. 


sk) (B®, cw} :P _, Qh) (2.4) 


Each of the elements pa in P“) and "i in Q“ is a vector containing 
the object properties. The Kalman filter is typically used for a multi-sensor 
data algorithm that employs Bayesian rules for the noisy environmental sensor 
measurements to produce reliable estimates of unknown quantities. 
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2.4 Retrospective and Prospective Safety 
Evaluation 


The vehicle-based safety evaluation can be categorized as retrospective and 
prospective. The major difference is the timing of the assessment with regard 
to the life cycle of the development process. Real accident databases are used 
for the retrospective analysis. In contrast, prospective analysis estimates the 
number of critical situations that may occur in the future [FSLL19]. In a given 
ADF, human drivers encounter an average number of kilometers between 
events as a benchmark of human performance. The human-driver failure rate 
is assumed to have a Binomial distribution. 


A safety case exists when the system under test has a failure rate lower than 
or equal to the benchmark reference with a particular level of confidence. 
Therefore, the number of test kilometers required for statistical evidence of an 
automated driving system can be calculated by a benchmark reference for the 
expected interval between accidents of equivalent severity. The total fatality 
rate in Germany caused by heavy-duty trucks in 2015 was 787 fatalities — as 
indicated in chapter 1 — , totaling 58, 93 billion kilometers [DES16]. Accord- 
ing to the Binomial distribution, the equation 2.5 represents the confidence 
level C[%] for an ADF with m failures during a cumulative driving distance 
de [km]. 


m de! _ 
Cian l= y ——_ 4 aay (2.5) 
as ran 


If the failure rate of a CMV is A[1/km], then the reliability y[1/km] is (1—4) 
and can be interpreted as the probability that there is no failure in the route 
driven. A hypothesis about the scenario (failure-free driving) can be used to 
estimate a lower limit for the number of failure-free kilometers to determine the 
reliability of automated CMVs with a confidence level C[%]. Consequently, 
the safety can be claimed for a certain number of failure-free kilometers at a 
particular confidence level, as shown in equation 2.6 [KP16]. 


Cig) =1-(1-2)% (2.6) 
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Equation 2.7 gives the required driving distance d.[km] without failures for 
given confidence C[%] and reliability y[1/km]. 


In (1 = C(¢=0)) 

ce = —— 2.7 
In(1-4) el) 

The required driving distance d.[km] is calculated by substituting the failure 
rate A with —84_, = 1.34 « 1078 [1/km] and the confidence level C with 


© 58.934 109 : 
95%, as indicated in equation 2.8. 


In (1 — 0.95) 


2 290) x 10°%km 28 
in (1 - (1.34 « 10-3) i i 


de = 


Figure 2.7 depicts the failure rate factor (AA+Ap), where 1,[1/km] is the 
failure rate of an ADF and Ay[1/km] is the benchmark failure rate of a human 
driver. For the CMVs these days, such long distance validations at which the 
controllability of the driver provides the necessary Proof of Safety (PoS) is un- 
necessary. However, in case of a fully automated driving system, the 2 million 
kilometers used to validate the current driver assistance systems are sufficient 
to prove the fatality factor A. The fatality factor A (2#10°km) is 25.5 times 
that of the humans, with about 50% confidence. Moreover, about 340 million 
kilometers are needed to prove that an automated driving system has a failure 
rate similar to that of human drivers in 2015 as the benchmark failure rate. This 
is done assuming that CMV has no failure (m = 0) during the FOT, with 99% 
confidence level. For this reason, it is economically impossible to demonstrate 
the safety of automated driving systems with widespread usage statistically 
before introduction, defined as an approval trap. 


While the critical traffic events are typically rare and not reproducible, early 
identification of functional deficiencies is essential for automated driving. 
Despite the difficulty of predicting all possible operating scenarios a priori, 
the coverage of critical driving scenarios needs to be adequately investigated. 
Recent research suggests the hypothesis of Poisson distribution to calculate the 
required validation distance with the following assumptions [Wac17, KP 16]. 
The Poisson distribution is a discrete probability distribution that expresses the 
probability of a given number of events in a continuum of time or space. Here, 
the route used is representative, while the critical events occur independently 
of each other within a random process. 
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Figure 2.7: Prospective failure-free kilometers for a failure rate factor compared to human-driven 
CMV fatality rate of the year 2015 [ESO*19]. 


In the equation 2.9, m corresponds to the number of accident events and 
A[1/km] is the predicted distance at which this event occurs at a given confi- 
dence level. 


A£ 
Citen) = =, © 3 $ =0,1,2, + ,m (2.9) 


The MTBF can be determined at a given confidence level using the hypothesis 
of the Chi-square distribution according to ISO 26262:2018. The Chi-square 
distribution is a probability density function that calculates the MTBF failure 
rate based on observed failures. Accordingly, an exponential failure distribu- 
tion with a constant failure rate is assumed. Regarding the safeguarding of 
driver assistance systems, there are no legal requirements for the validation 
distance. 
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Since unintended reactions are rare events, a Chi-square distribution can be 
applied. If no critical event occurs at a sample distance with a required failure 
rate of one million kilometers each, the necessary validation requires around 
three million kilometers. In this case, no event should occur during the driv- 
en interval to argue the residual risk with a confidence level of 95%. The 
required mileage will increase if more events occur during validation (e.g. 
de= 4.8 * 10°[km] at C= 1, de= 6.3 + 10°[km] at {= 2, de= 7.8 + 10°[km] at 
C= 3, etc.), as illustrated in figure 2.8. In practice, the validation distance does 
not play the central role, but the variance of test conditions do, in order to cover 
maximum possible rate operating situations (e.g. different weather conditions, 
time of day, road conditions, traffic conditions, pedestrian conditions, etc.). 
Therefore, route diversity in physical road tests is a significant measure of the 
probability distribution. 


failure rate (A)[*10~°/km] 


0 5 10 15 20 
cumulative distance (d.) [*10° km] 


Figure 2.8: Required validation distance for various accident events using the Chi-square distribu- 
tion with confidence level (C = 95%) [ESO*19]. 
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However, statistical evidence of the accumulated road kilometers is potentially 
invalid with each software upgrade. Even if the FOT continues in the Spiral 
model of software development until no more errors are found, the safety case 
argument does not provide any proof that the ADF is absolutely safe due to 
the Pesticide paradox phenomenon [Bac18]. In software testing, the Pesticide 
paradox is an error detection phenomenon, where if the same test matrix is 
performed repeatedly, the same test matrix will eventually find no more errors 
[KW 18]. It means that an automated driving algorithm that passes the same 
repetitive tests eventually builds up resistance to them. Consequently, it will 
not be viable to prove safety of the required level of system performance 
through driving test hours alone during the development phase. Furthermore, 
there is no complete public set of machine-interpretable traffic regulation with 
exception-handling rules. 


The functional requirements for ADFs become thus implicit and incomplete by 
the learning process from machine learning data-sets, which are used to perform 
algorithmic operations. The reliance on data-driven mileage accumulation as 
the only credible safety argument points to an impractical safety validation 
strategy. Also, the real-world testing may not accumulate enough hours of 
exposure to observe critical scenarios that occur by chance. On the other hand, 
knowledge-based assessments can accelerate exposure to the known critical 
scenarios but suffer from the possibility of not verifying the unanticipated 
scenarios. 


Alternative methods of safety assessment are therefore required, as the val- 
idation distance in the FOT will increase dramatically by using the current 
test concepts for automate driving without driver engagement. It is therefore 
obvious that a safety argument for these algorithms from SAE L3 onwards, 
based solely on the accumulation of road kilometers through endurance test 
campaigns, is no longer feasible. Therefore, knowledge-based test platforms 
can fulfill the associated test objectives in a time and cost efficient manner. 
However, these techniques alone cannot guarantee a sufficient confidence of 
safety for large-scale deployment without giving particular attention to data 
acquisition and analysis from the FOT [Z* 18b]. 
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2.5 Measurable Safety Methodology 


The functional safety is an absence of unreasonable risks due to hazards caused 
by malfunctioning behavior of E/E systems. Therefore, the ISO 26262:2018 
norm is a risk-based approach for the development of safety-critical software 
systems in passenger cars, motorcycles and commercial vehicles. The ISO 
26262:2018 includes a hazard analysis and risk assessment to determine the 
required Automotive Safety Integrity Level (ASIL) and to assess the potential 
risks of E/E malfunctions that may violate the safety goals. The risk-oriented 
approach classifies the risk R for each potentially hazardous driving situation, 
as shown in equation 2.10 [Hun18]. 


heH 
Rw > (Sh * Xh * On) (2.10) 
0 


The hazard classification defines each potentially hazardous driving situation 
with the following three impact factors, where H corresponds to the set of 
hazardous driving situations [HKM17]: 


e Severity level Sp of a hazardous driving situation h with the consequence 
of injuries or fatalities, where Sp € { Ss? , Si ; S? , Ss +, 


e Exposure probability level X, of hazardous driving situations, where Xp 
Sia A, ee 


e Controllability probability level O, of not avoiding an accident in a 
harmful situation, where On € { O° , ol , O? , Oo} +, 


In this context, the system follows one of five classes to define risk reduction 
requirements, where ASIL D is the highest and Quality Management (QM) 
the lowest risk reduction class (ASIL 26262-1:2018). For example, a system 
specified for the implementation of a truck platooning may exhibit undesirable 
behavior due to a misclassification of objects and require driver intervention. 
Therefore, controllability presents the probability of controlling driving situa- 
tions within system limits and failures. The processes and methods for assessing 
the controllability of unintended driver assistance reactions are specified in the 
Code of Practice (CoP) for the design and evaluation of human assisted driving 
systems [KNB*09]. 
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The ASIL determination can be assigned to the level ON of controllability for 
automated driving without driver intervention, where the intended function is 
difficult to control or uncontrollable, as illustrated in table 2.5 [KW16]. 


I I IE 222 sls ss & 
ASIL level 

N N AA OE | ae 2 8 Xe 
QM ©e © O O|J|® O O 0/0 O 0.0 
ASIL A O O © ea, 0 © O 0/8 O O O 
ASIL B O O O 0/0 O @® 0/0 0 0 0 
ASIL C O O O 0/0 O O @®/O O 0 O0 
ASIL D O O O 0/0 O O 0/0. O O © 
©: relevant 


O: irrelevant 


Table 2.5: ASIL requirements for automated driving devices without driver monitoring using 
uncontrollable level o according to ISO 26262. 


Despite the updating of the scope of the ISO 26262:2018 norm for the inclusion 
of CMVs in Edition 2, its safety goals mainly address undetected random 
hardware failures of the system components and systematic software failures 
[SH19c]. Assuming that the E/E system malfunctions are managed using ISO 
26262:2018, the safety violations, which may be caused by the environmental 
perception sensors, remain outside the scope [SH20]. Redundancy, diversity 
and functional restrictions can compensate system limitations [SH19a]. 


The ISO/PAS 21448:2019 serves as an extension scheme to specify the in- 
tended function in such a way that it is robust and safe enough to take the 
variations in sensor inputs and the different environmental conditions into 
account [fS*19b]. The OEDR examines whether the vehicle can correctly 
detect objects and events and execute an appropriate response. Therefore, the 
context of the OEDR is similar to the case in SOTIF, but with a different 
designation given by the NHTSA. However, new verification and validation 
measures are needed to assess unintended system behavior due to technological 
and systemic deficiencies. Atthe same time, the ISO/PAS 21448:2019 activities 
complement the ISO 26262:2018 norm with its focus on driver assistance rather 
than automated driving without driver engagement. 
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On 22. Mai 2022, the German Federal Council (Bundesrat) approved an ordi- 
nance that regulates the operation of motor vehicles, in particular CMVs, with 
SAE L4 ADFs (Verordnung zur Regelung des Betriebs von Kraftfahrzeugen 
mit automatisierter und autonomer Fahrfunktion und zur Änderung straßen- 
verkehrsrechtlicher Vorschriften) on public roads within a specified operating 
range [Bun22]. The ordinance supplements the German act on autonomous 
driving (Gesetz zur Änderung des Straßenverkehrsgesetzes und des Pflichtver- 
sicherungsgesetzes — Gesetz zum autonomen Fahren) and establishes a legal 
framework for the safe deployment of CMVs with a SAE L4 ADF for operation 
on public roads in Germany [KML22]. In addition, the ordinance introduces a 
uniform procedure for the approval of tests by the German Federal Motor Trans- 
port Authority (Kraftfahrt-Bundesamt) in order to standardize and centralize 
the tests pursuant to §1i of the German Road Traffic Act? (Straßenverkehrsge- 
setz — Erprobung von automatisierten und autonomen Fahrfunktionen), with 
the provision of a safety concept for functional safety. 


2 https: //www.gesetze-im-internet.de/stvg/BJNRO04370909.html 
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Perspectives 


The systems engineering process requires the state-of-the-art evaluation pro- 
cedures to verify and validate ADFs [SZS*15]. The evaluation of software 
releases needs to be carried out in different phases up to the start of production 
to prove that the residual risk is below an acceptable level. Initially, the driving 
simulator tests are used to evaluate various system concepts of a CMV [Has 14]. 
Testing programs are carried out in XiL, on test tracks and in environments un- 
der real traffic conditions, as already defined in subsection 2.1.7. Several tools 
are used in this context including the following: Hardware/ Software (HW/SW)- 
open-Loop reprocessing, HW/SW-in-the-closed-Loop simulation, customer 
studies in driving simulator as well as real-world test drive open-loop and 
closed-loop, as already defined in subsection 2.1.3. The scenario databases 
are then integrated into the data analysis and assessment tools. Therefore, 
the induced traffic situations with unintended reactions continuously extend 
the scenario databases. Finally, homologation is used to indicate that the 
approval requirements for the market-specific type are met, based on the evi- 
dence collected during the development process [R* 18a]. Appropriate quality 
measures are essential to achieve sufficient reliability, safety and availability 
in the framework of the software quality management process. 


3.1 Data-driven and Knowledge-based Test 
Platforms 

The data-driven test methods use empirical data to obtain new insights into the 

system behavior under specific traffic situations or field conditions, while the 


knowledge-based test methods convert the implicit information into explicit 
ones. 


59 


3 State-of-the-Art and Research Perspectives 


Data-driven test methods require many prerequisites such as fleet vehicles 
equipped with additional high performance measurement systems and data 
processing pipelines. Meanwhile, knowledge-based test methods use abstract 
information to create functional, logical or even directly concrete scenarios for 
the database. The information can be in the form of abstract knowledge from 
experts, standards and guidelines. Table 3.1 illustrates various data-driven 
and knowledge-based test methods (e.g. New Car Assessment Programme 
(NCAP) tests, requirements test metrics, FOT and scenario databases) for safe- 
ty and reliability assessments [ZKK*16]. Although ISO 26262:2018 and its 
V-framework reflect generally accepted practices to ensure functional safety, 
ADFs present unique challenges in mapping the technical and functional re- 
quirements to the classical V&V methods [KW17]. Therefore, the validation 
procedures offer a range of activities to generate confidence that an ADF can 
achieve its intended purpose and goals. 


Scope Inference 
Data-driven testing Knowledge-based testing 
(induction) (deduction) 
Case-by-case Empirical data System use cases 
analysis (e.g. NCAP tests) (e.g. requirements test matrix) 
Traffic-based Route profile Scenario meta-model 
evaluation (e.g. on-road field tests) (e.g. scenario databases) 


Table 3.1: Classification of data-driven and knowledge-based test methods for safety and reliability 
assessment [ESm*19] 


3.1.1 HW/SW-in-the-open-Loop Simulation 


Figure 3.1 depicts the general data flow of a regression test using the example 
of an eliminated software error within the object detection of a monocular 
camera sensor. The relevant situations for the camera ECU are recorded by the 
Ego-vehicle with an appropriate measurement equipment and collected within 
a data ingestion process. A software update occurs within a software develop- 
ment process. While the environmental perception sensors react sensitively to 
the target hardware constraints, the monocular camera sensor without camera 
optics is integrated for regression tests with recorded sequences. 
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This is followed by the Hardware-in-the-open-Loop (HoL) test bench stimu- 
lating the optical interfaces of the monocular camera sensor with the recorded 
data to verify the functionality of the new software release [WRM* 19]. 


Definition 3.1 (Open-Loop Reprocessing): It verifies the error correction by 
reprocessing of field measurements with new software releases. In the case of 
software reprocessing, the target software is executed on prototypical hardware, 
whereby the software decisions have no influence on the stimulus. In the case 
of hardware processing, the target software is executed on the target hardware, 
while the hardware outputs have no influence on the hardware inputs. 


Definition 3.2 (HoL Test Platform): A Hardware-open-Loop reprocessing 
platform to enable reprocessing of the target software on the target hardware, 
while the hardware outputs have no effect on the hardware inputs. The test 
platform is typically conducted within a validated environment for components 
and subsystems of the environmental perception. 


Therefore, the HoL test bench stimulates the external interfaces in real-time 
by utilizing a recorded sequence of on-road tests with an original detection 
result (v) to generate a new detection result (V) after updating the application 
software of the target hardware, as illustrated in figure 3.1. Thus, original and 
new results can be compared to decide whether the error is fixed according 
to defined pass/fail criteria. The hardware open-loop reprocessing generates 
driving situations from the test-case description and evaluates the behavioral 
response without feeding it back into future situations. 


Figure 3.1: Regression test process with the HoL test bench using the example of detection of 
oncoming objects arel (left) with a monocular camera sensor (right) [E*16]. 
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3.1.2 HW/SW-in-the-closed-Loop Simulation 


The HW/SW-in-the-closed-Loop simulation verifies the behavior in a closed- 
loop to prove functional correctness of an artifact against its functional 
specification. A scenario-based testing requires various technical requirements 
for the simulation environment of roads, traffic objects, environmental per- 
ception sensors, the driver of the Ego-vehicle, commercial vehicle dynamics 
and actuators. For a camera HiL test bench, the traffic environment shall be 
animated in 3D perspective to display the traffic scenes in a virtual world in the 
form of a video sequence recorded by a monocular camera sensor, as illustrated 
in figure 3.2. Therefore, closed-loop testing specifies an entire scenario in a 
test case that contains a sequence of scenes, actions, events and goals for the 
ADF. The behavioral reactions are used to influence future scenes and thus 
also future situations. The HiL test method integrates the ADF into the traffic 
environment and vehicle dynamics simulations by combining the simulation 
models and the ECU hardware into a real-time ECU test bench. However, the 
Software-in-the-Loop (SiL) test platform integrates the executable software 
code generated from the same source as for the automotive ECU. 


Definition 3.3 (HiL Test Platform): It refers to a test bench to enable process- 
ing of the target software on the target hardware, with the hardware outputs 
influencing the hardware inputs. The test platform can be used either at com- 
ponent, subsystem or system level. 


The monocular camera ECU is connected to the HiL via a Controller Area 
Network (CAN) bus for rest-bus simulation purposes [TH13]. By the means 
of a monitor positioned in front of the camera, the camera is stimulated to 
pretend reality by processing the displayed realistic images under real-time 
conditions. Figure 3.2 illustrates an example with a lane change scenario, which 
is applied to a LDW function. As an objective of benchmarking, the reference 
values are compared with the detected values by the camera projection of 
the monocular camera ECU within the real-time simulation environment; e.g. 
dy [m] and uy [m/s] for the left lane and dy [m] and Uy [m/s] for the right lane. 
In chapter 4, a detailed approach to HiL techniques, with their implementation 
is discussed. 
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Figure 3.2: Verification process with the closed-loop HiL test bench using the example of the 
detection of left lane markings (left) and right lane markings (right) on the basis of a 
monocular camera sensor [ESW+16]. 


3.1.3 Model-based Back-to-Back Testing 


Model-based software development in the automotive industry uses tools such 
as Simulink®or dSPACE TargetLink to implement a software module within 
the ADF. Simulink®is a graphical programming environment for modeling 
and simulating of automotive dynamical systems. The dSPACE TargetLink 
is a tool for automatic C and C++ code generation based on a subset of 
Simulink®/Stateflow®models. Tools such as Simulink Coder™ or dSPACE 
TargetLink are then used to automatically generate C source code from the 
resulting models. Simulink Coder’ is a tool for automatic code generation from 
Simulink® diagrams, Stateflow®charts and MATLAB® functions. Automated 
test data generation can be applied if a test oracle can be defined automatically 
with its reference information. 
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Definition 3.4 (Test Oracle): A mechanism in software testing to determine 
whether a test case has passed or failed. For example, structural testing is 
typically employed to generate test data based on the internal structure of the 
test object. Therefore, the identification of input values depends on a selected 
path or branch that is executed within the test object. 


The ISO 26262:2018 demands that coverage metrics shall be taken into account 
when testing at the model and software code level, such as MC/DC. A major 
cause of such semantic differences is the application of scaling to variables 
during the software code generation to optimize code efficiency and value 
precision. In the figure 3.3, back-to-back tests generate a collection of structural 
test cases to compare the software generated with the behavior of the underlying 
model. 


Definition 3.5 (Back-to-back Consistency): It is a type of software testing. 
Two or more variants of a software module are generally tested with the same 
stimulus inputs. Their corresponding outputs are compared and evaluated if 
there are discrepancies in the software. 


Figure 3.3 illustrates an example for defining a test oracle to evaluate the 
automatically generated test cases according to the accepted time and value 
tolerances. Therefore, both, the model and software are executed with the same 
input data and then the corresponding output data entries are compared. The 
value tolerance is determined by the difference between the vmi and vsi of 
the longitudinal acceleration a,“ [m/s”] within an ACC function, as illustrated 
in the left side of figure 3.3. In contrast, the time tolerance is determined 
by the difference between the vmi and vsi of the required mode rollmode 
within an ACC function, as depicted in the right side of figure 3.3. The model 
runs with the Model-in-the-Loop (MiL) test suite and is verified back-to-back 
with the C code running on the SiL test suite. Wilmes introduces a hybrid 
test data generation approach to combine static analysis and dynamic test 
data generation [Will5]. The test data finding problem is converted into an 
optimization problem by defining a cost function. As a result, the generated 
test data is evaluated to distinguish between relevant and irrelevant test data and 
to generate new test data in each iterative cycle [Wil16]. In addition, the static 
analysis serves to accelerate the automatic search by identifying unreachable 
model states. 
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Figure 3.3: Back-to-back test process with the model-based code generator (ASPACE TargetLink) 
using the example of an ACC function based on value tolerance (left) and time tolerance 
(right) [ESO* 19]. 


3.2 Goal-based Safety Case Assessments 


The ISO 26262:2018 standard represents the state-of-the-art with respect to 
functional safety for safety-critical E/E systems in road vehicles. The CoP for 
the design of human assisted driving includes the evaluation of the human 
assisted driving functionality and is based on the driver’s controllability to 
maneuver the CMV’s reliability in road traffic [KNB*09]. These practices 
assume that the human driver remains responsible for CMV behavior to over- 
ride or deactivate the system at any time. If the driver is no longer responsible 
for the behavior of the CMV, which is already the case with intervening 
emergency functions, the driver’s controllability test catalogs are no longer 
sufficient. The evidence shall also be provided that the Type I and Type Il rates 
are reasonably low [AW17]. 
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The approaches of safety validation for ADFs beyond the mileage accumulation 
are in high demand. Thereby, a falsification approach shall be coupled with 
concrete, verifiable safety objectives and requirements. In parallel, the verifica- 
tion procedures according to the ISO 26262:2018 V-shaped model assume that 
high-quality requirements for implementation are further developed. There- 
fore, the traditional V-shaped model engineering process can pose a challenge 
in articulating the functional requirements of machine learning algorithms 
[KW16]. With the V-shaped model, the training set is more related to the 
functional requirements and the validation set to a test plan. The verification 
arguments with sufficient training and validation data leads to the need to de- 
velop the data ingestion system according to safety-critical software standards 
[Hill2]. 


The American National Standards Institute (ANSI)/ Underwriters Laboratories 
(UL) 4600 is a standardization activity to address the ability to autonomous 
products to perform the intended function without human intervention. This 
is based on their current state and sensing the operating environment. The 
standard intends to apply a goal-based approach that specifies tasks that need 
to be addressed in creating a safety case [UL22]. The safety case shall argue 
that relevant objects from existing sensors can be successfully detected and 
classified within the intended ODD, rather than determining whether a system 
design with a new sensor setup is required. Meanwhilst, the safety case should 
provide an argumentation to ensure an appropriate safety level through arobust 
combination of analysis, simulation and testing, rather than deciding how many 
kilometers have to be accumulated for safety demonstration purposes [KW 17]. 


The Goal Structuring Notation (GSN) presents a safety case to highlight 
the verification methods for automated truck driving. The GSN is a graph- 
ical structure notation for structuring an assurance case in connection with 
argument, context, assumptions and evidence. The safety case is a reasoned 
and verifiable artifact that supports the contention that its top level claim is 
satisfied, including systematic reasoning, its underlying evidence and explic- 
it assumptions that support the claim according to ISO/IEC 15026-2:2011. 
Therefore, the assurance case ensures that sufficient evidence is systematically 
gathered to argue a tolerable residual risk through adaptive verification for 
ADFs, as illustrated in figure 3.4. Each hypothesis identifies the residual risks 
for a test or simulation environment. The assumptions that are covered by other 
verification approaches are a part of the safety argumentation chain [BGH17]. 
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Figure 3.4: Argument structure of context-driven test concept based on the ODD coverage using 
GSN [ESO*19]. 


The goal-based safety case approach follows a decomposition approach on 
functional, logical and concrete levels. Therefore, the test scenarios can be 
described either as functional with NL description without values, logical with 
an assignment of value ranges or concrete with an association of fixed values. 
A major challenge in achieving the decomposition approach is to classify the 
test objectives and the coverage criteria according to their respective test en- 
vironments [Ame20]. Therefore, the test objectives imply measurable quality 
criteria for the V&V strategy. 
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Strategy (S/):It describes the verification case strategy to define the required 
termination criteria for testing through adaptive test coverage with the con- 
text elements {C5,C6}. Subsequently, the following sub-goals {G2,--- , G7} 
provide the evidence arguments {E/,--- , E6} of test coverage within the 
verification case [GMB 18]. 


Goal (G/):It is the main goal to constitute the top-level claim of the verifi- 
cation case scope with the context elements {C/,--- , C4}. The G/ argues 
a sufficiently low level of residual risk associated with individual hazards in 
automated driving, although all possible driving situations might not be verified 
during the development phase, detailed in section 6.1. 


Goal (G2):The proof of functional correctness verifies that the test object 
fulfils the required functionality according to its specifications by leveraging 
synthetic and real-world data, discussed in subsections 3.1.2 and 5.3.3. 


Goal (G3):The proof of back-to-back consistency is the verification of the 
required consistency between the various execution platforms (e.g. model 
and code) within the permissible discrepancies by back-to-back tests for 
algorithmic-based software structures, illustrated in subsection 3.1.3. 


Goal (G4):The proof of system integration and variation is the evidence to 
the validity of the system maturity to cover the system variations that may 
include static and dynamic tolerances of truck-trailer combinations, expound 
in subsections 4.2.1 and 4.2.2. 


Goal (G5): The proof of software robustness shall provide a sufficient probabil- 
ity of coping with system boundaries and faults using fault injection techniques, 
as explained in subsections 3.3.2 and 3.3.3. 


Goal (G6): The proof of sensor availability and functional effectiveness focus- 
es on the presence of the environment perception sensor within the defined 
deviation and tolerance limits of the specified time and range using regression 
tests and re-simulations, as focused within subsection 3.1.1. 
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Goal (G7): The proof of the software reliability and functional safety is the 
proof that the test object is reliable enough with respect to functionality and 
safety mechanisms by leveraging field observations with FOTs and synthetic 
data. The proof of functional safety refers to the functional safety requirements 
of ISO 26262:2018 in order to avoid systematic software and random hardware 
failures, as portrayed in sections 6.2, 7.1, 7.2 and 7.3. 


Table 3.2 explains the assignment of different possible test environments with 
the respective test objectives. The selection of suitable test environments from 
XiL to FOT depends on the effectiveness and efficiency criteria of the test 
conditions and their validity. The effectiveness criteria indicate the intended 
results such as representative valid and observable interfaces. However, the 
efficiency criteria reflect the desired performance in comparison with the re- 
sources used to achieve the economic use, reproducibility and promptness. The 
proving ground is an area dedicated to putting a vehicle’s performance to the 
test. In the driving simulator, the driver is in an artificial environment designed 
to replace one or more aspects of real driving behavior. 


Test environments Test objectives 


MiL tests 

SiL tests 

Component HiL tests 
Subsystem HiL tests 
System HiL tests 
Proving ground tests 


Driving simulator studies 
Naturalistic FOTs 
HW/SW reprocessing & regression tests 


© 0 00000 ©) ©) 7 
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@ : recommended test objective 
O: not useful 


Table 3.2: Assignment of potential test environments to the corresponding test objectives — driven 
from figure 3.4 — [ESO*19]. 
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Table 3.3 explains the assignment of different possible test suites with the 
respective test coverage criteria. The test coverage criteria provide an indicator 
of the software testing effort during a test run within the V&V strategy. 


Test environments Test coverage criteria 


MiL tests 

SiL tests 

Component HiL tests 
Subsystem HiL tests 

System HiL tests 

Test tracks & proving grounds 


Driving simulator studies 
Naturalistic FOTs 
HW/SW reprocessing & regression tests 
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CO} IK IKK IK @} o|o o% 
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o| @| @ ol Oo) e| @) Ofc) 3 


@ : recommended test coverage 
©: not useful 


Table 3.3: Assignment of possible test environments to the corresponding test coverage criteria — 
driven from figure 3.4 — [ESO* 19]. 


Evidence (E/):The functional requirements coverage defines a relationship 
between the functional requirements and the executed test cases, whereby at 
least one test case is defined for each requirement. 


Evidence (£2): The software structure coverage provides the code coverage of 
model-based software structure components, such as MC/DC. 


Evidence (£3): The system integration coverage includes detected failures in 
the interfaces and interactions between integrated components, subsystems or 
systems. The system variation coverage defines the robustness against varia- 
tions in the system context. For example, when automated driving software 
modules are developed, a variety of system variants which can include static 
and dynamic tolerances within the Ego-vehicle, are put in. 


Evidence (£4): The software performance coverage determines the robustness 
of the software using fault injection techniques. 
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Evidence (E5): The training data coverage specifies which training data is re- 
quired for a particular application and which data leads to the most accurate 
results, such as training of neural networks for image processing. The uncer- 
tainty coverage quantifies the Aleatoric and Epistemic uncertainties of machine 
learning algorithms. 


Evidence (E6): The driving scenarios coverage identifies the known critical 
scenarios, which should exhibit similar behavior, and minimizes unknown 
critical scenarios. 


3.3 Standardization Activities and Research 
Projects 


There are various standards that determine the assessment of SAE LO-L2 
functions for commercial vehicles and buses based on the physical characteris- 
tics of the vehicle. The ISO 19377:2017 describes a test method for determining 
the path deviation of the braking maneuvers induced by an AEB of CMVs from 
a predefined desired trajectory. If the AEB function is utilized in further stages 
of the automated driving systems, the deviation needs to be compensated. 


The ISO 15622:2018 presents the minimum requirements for failure reactions 
and performance test procedures for ACC systems. The ISO 22839:2013 ap- 
plies criticality assessment metrics on an AEB system to provide a certain ratio 
of Type I and Type II errors within the receiver operating characteristic curve 
space. The ISO 22839:2013 is a test method standard for defining the required 
behaviors and test criteria of an AEB system. Shladover et al. [SN19], Takacs et 
al. [TDG*18] and Junietz et al. [I[WKW18] provide a comprehensive overview 
of the activities concerning regulations and standards for the type approval of 
automated vehicles. These standardization activities can be considered as de- 
velopment guidelines for manufacturers. In recent years, many research projects 
have been completed dealing with the V&V activities for SAE L3 automated 
vehicles. The following is a summary of representative completed projects: 


e The research project PEGASUS (Project for Establishing Generally 
Accepted good quality criteria, tools, methods, Scenarios and Situations 
for the approval of highly ADFs) aimed at developing methods for en- 
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suring the safety of ADFs [N*20]. The aim of the research project was 
to develop uniform technical standards for the verification of condition- 
al ADFs, and to define thresholds for a sufficiently high controllability 
level of SAE L3 systems [Sch15]. The PEGASUS! project has seeked to 
define risk scenarios, assessment criteria for sufficient safety in various 
traffic situations on the basis of a generally accepted method for evaluat- 
ing the safety of conditional ADFs [WLFM19]. The project results were 
demonstrated in the middle of 2019 on the basis of a highway chauffeur 
as a SAE L3 system for passenger cars [M*16]. 


The research project L3Pilot? (large-scale Piloting of SAE L3 functions 
on European roads) was concerned with large-scale field tests of SAE L3 
functions under variable conditions with 1000 drivers and 100 vehicles 
[HSK*19]. The field tests served to evaluate the technical aspects, user 
acceptance, driving behavior and safety impacts for SAE L3 applications. 
With the comprehensive piloting of ADFs in test vehicles, L3Pilot has 
paved the way for large-scale field tests of series cars on public roads. 


The research project AdaptIVe? (Automated driving applications and 
technologies for Intelligent Vehicles) was an European research project 
dealing with safety validation, technical system limits and legal aspects 
ofthe release of automated driving applications. The project results have 
presented best practices in systems engineering and safety validation to 
establish a CoP for the design and evaluation of ADFs [Ete17]. 


The research project SaLsA (Safe autonomous Logistics and transporta- 
tion vehicles in outdoor Areas) focused on developing automated trucks 
that can operate safely outside warehouses in a shared work environment 
with conventional human-driven vehicles and pedestrians. The coopera- 
tive scanning of the environment through mobile and stationary sensors 
enables safe operation even at higher speeds [Wie17]. 


The research project ATLaS* (AuTomated and networked driving in 
Logistics - opportunities for more added value) investigated the influ- 


https: //www.pegasusprojekt.de/en/home 

2 https: //www.13pilot.eu/ 

3 https: //www.adaptive-ip.eu/ 

4 https: //www.tib.eu/de/suchen/id/TIBKAT:1697842658/ 
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ences of automated and connected driving on the logistics chain in order 
to identify deployment scenarios accepted by stakeholders [FL*20]. 


The research project ENABLE-S3° (European initiative to ENABLE 
validation for highly automated safe and secure systems) was a research 
project to enable an accelerated assessment of highly automated and 
autonomous systems in the mobility domains (e.g. automotive, avion- 
ics, rail and maritime) [LAH*19]. The project objective is to establish 
cost-efficient cross-domain virtual and semi-virtual V&V platforms and 
methods for autonomous cyber-physical systems [Lei20]. 


In parallel, numerous research projects have been launched to contribute to 
developing a general consensus or uniform framework for safety validation 
and reliability assessment of SAE L4-L5 automated driving. The representative 
ongoing projects in industry and science to safeguard automated driving are 
summarized as follows: 


The test field TAF-BW® (Test Area Autonomous Driving Baden-Wiirt- 
temberg) offers a testing environment under real traffic conditions for 
automated and connected driving applications. The test area is funded 
by the state of Baden-Wiirttemberg and includes all relevant road types. 
The aim of the test area is to promote the development of future- 
oriented solutions for individual traffic and local public transport. The 
ground truth data is derived from the infrastructure sensors for real-time 
recording of traffic and its influencing factors as well as from 3D HD 
maps of the road network in everyday road traffic [FDW* 18]. 


The research project VVM” (Verification and Validation Methods for 
SAE L4 and L5 automated vehicles) aims to develop procedures to 
combine the virtual and the real-life tests. The developed procedures can 
be used for legally compliant and efficient homologation of automated 
vehicles [ZRBE20]. 


The research project SET LEVEL 4t05® (Simulative dEvelopment and 
Testing of LEVEL 4 and 5 systems) addresses the simulation-based 


5 https: //enable-s3.eu/ 

6 https: //taf-bw.de/en/ 

7 https: //www.vvm-projekt.de/ 
8 https://setlevel.de/ 
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development and testing of SAE L4 and L5 vehicles for urban areas. 
The project intends to provide tool-oriented techniques for efficiently 
designed simulation-based test and release procedures [HTR*20]. 


e The research project KI-Absicherung? (Methods and measures to safe- 
guard artificial intelligence-based perception functions for automated 
driving) seeks to develop measures and methods to validate neural net- 
work based perception and multi-model sensor fusion. The pedestrian 
detection is identified as a representative function for the project in terms 
of multi-sensory perception [GGSB19]. 


In addition, a significant number of publications on safety validation of 
automated vehicles have been published in recent years in response to the 
strong interest in a rapid market introduction of automated vehicles. Table 3.4 
refers to a comparison with a scale from not applicable to optimal using the 
following evaluation criteria. According to the current state of science and 
technology, the diverse approaches of safety assessment for automated driving 
can be divided into four categories: shadow-mode approach, formal safety 
verification, traffic simulation-based approach and scenario-based approach, 
as described in the following subsections 3.3.1, 3.3.2, 3.3.3 and 3.3.4. 


First, the representativeness of scenarios reflects how realistic road traffic 
conditions can be used. Second, efficiency in the identification of corner 
cases is of crucial importance for system developers. Third, the scenario 
space coverage represents how the permutation coverage is achieved within the 
physically possible parameter space. Fourth, the safety assessment approaches 
can be distinguished based on the system applicability for perception, pre- 
diction and planning modules. Fifth, the computational feasibility shows the 
applicability of the assessment approach for ADFs, e.g. truck platooning. Sixth, 
the reliability of statements indicates which evidence exists for the safety ar- 
guments. Then, the extrapolation of risk metrics illustrates the scalability by 
transferring the microscopic assessment results to macroscopic assessment. 


Next, the modeling dependencies define the model validation efforts to argue 
the safety statement. After that, the dependence on safety drivers points to 
the challenges to launch ADFs under supervision of safety drivers. Finally, 
the closed-loop interactions define the approach validity for predication and 


9 https: //www.ki-absicherung-projekt.de/ 
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planning modules in which future trajectories of other road users are influenced 
by the automated vehicle’s behavior. 


Evaluation criteria Safety assessment approaches 


Representativeness of scenarios 


Identification of corner cases 


Scenario space coverage 


System applicability 


Computational feasibility 


Reliability of statements 


Extrapolation of risk metrics 


Modeling dependencies 


Dependence on safety drivers 


©! ©0006 @| O| O| CO} Traffic simulation-based approach 


Oo @| O| ©} O O ©! ©} @| O| Shadow-mode approach 
@| O| O} OC} @| OO @| @| O| Formal safety verification 
©] O| © eo © @| ©! CO} @| Scenario-based approach 


Closed-Loop interactions 


©: optimal @: fairly optimal 
@: natural O: fairly poor 
O: not applicable 


Table 3.4: Comparable evaluation according to the state-of-the-art of categorized safety assessment 
approaches [SWB* 20, Jun19]. 


3.3.1 Shadow Mode Approach 


Wachenfeld proposes the use of stochastic methods for introduction of auto- 
mated driving without driver supervision. Here, the safety of these automated 
driving levels cannot be proven statistically using accumulated kilometers 
of physical onroad testing under consideration of an estimated uncertainty 
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[Wac17]. Wang and Winner [WW 19] describe an approach in which the ADF 
in end-customer vehicles is operated passively and is known as shadow mode. 
The passive function is equipped with the relevant sensory perception and 
logging devices, but does not have an access to the actuators [Wan21]. There- 
fore, a large fleet of human-driven vehicles can collect raw sensor data for 
HW/SW-open-Loop reprocessing. In addition, these open-Loop recordings are 
retrieved to validate prediction and planning algorithms, so that the recorded 
data can be used in a HW/SW-in-the-closed-Loop simulation environment. In 
this way, the simulation can be used to evaluate the decisions of the ADF in 
passive mode and thus determine the required safety level [Kar20]. One of 
the major advantages is the absence of safety risks during the data acquisition 
phase. However, one of the main disadvantages is that the behavior of the 
objects in the simulation does not correspond to reality because other road 
users also plan and execute their actions based on the actions of the active 
driving function [Wag21]. Therefore, the results of the simulation need to 
be argued. The car manufacturer (Tesla) announces to use the shadow mode 
testing to validate new ADFs and new software versions of existing systems 
[FBK17, Hull6]. 


3.3.2 Formal Safety Verification 


Althoff et al. [ASB07] introduced a reachability analysis which argues the 
safety of automated vehicles. Reachability analysis seeks to determine the 
states that a system can reach from given initial states and possible inputs 
and parameters [Alt10]. Through formal verification, a mathematical model 
is used to formally demonstrate the safety of trajectory planning across the 
entire ODD [AKM17]. The approach distinguishes between perceptual and 
planning concerns. Using a safety envelope, the trajectory planner model is 
defined as intrinsically safe [AL18]. The valid and explainable set of traffic 
rules is aligned with the employed model that restricts the actions of the actual 
trajectory planner and increases the transparency of the behavior planning. 


The IEEE 2846 is a standardization activity to define a formal rules-based math- 
ematical model for automated vehicle decision making using discrete mathe- 
matics and logic. Arechiga [Are19] defines a set of rules for automated driving 
and other road users that are formally verified using a formal language called 
Signal Temporal Logic (STL). The argument is that the entire traffic system 
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will be collision-free if all road users follow these rules. Loos et al. [LPN11] 
used a formal proof calculus for safety verification by proofing safety separately 
for adaptive cruise control cases applied in a distributed car-control system. 
Nilsson et al. [N*15] verified an AEB system using closed-form expressions 
for robust avoidance scenarios. The closed-form expressions are derived based 
on worst-case performance as an optimization problem between FP and TP 
interventions (elucidated in chapter 2). Since the model and its parameteriza- 
tion take a central role, the modeling is a key challenge for this approach. The 
concept of safety envelopes can be illustrated by a cut-out test scenario. In the 
cut-out scenario, a vehicle in front suddenly leaves the lane to avoid a stopped 
vehicle in its lane. 


The situation gives the AEB function a short time to detect and react accord- 
ingly. Figure 3.5 depicts a cut-out scenario in which the moving vehicle [obj2] 
is triggered at a cut-out distance d{° = 45[m] between the stationary vehicle 
[obj1] and the moving vehicle [obj2] to change to the adjacent lane after a 
cut-out delay time tco = 2[s]. 


safety envelopes : {no crash, crash} g : 
Pr obj2 
Aloe 

——... a y l 
Ego-vehicle se AR LER aE > i] 
ls driven path ^ bj2 LL obj 


stationary object 
Figure 3.5: Schematic diagram of a cut-out driving scenario with the safety envelopes. 


Figure 3.6 shows the pass/fail envelopes either as crash or no crash with (&], 
&? and &2) as escalation levels of the AEB function. The driving scenario with 
the different vehicles (ego, obj1 and obj2) is carried out on the HiL test bench 
and controlled by a test script based on a track co-ordinate system. 


The HiL platform includes perception sensor models for the RADAR and 
monocular camera sensors that are used within the AEB function. Therefore, 
the stationary vehicle [obj1] is occluded through the moving vehicle [obj2], 
where the Ego-vehicle and the moving vehicle have the same constant velocities 
on the same lane v\® = „oo = {30, 35, 40, 45} [km/h]. In the case of idealized 
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sensor models without occlusion effects, the cut-out scenario cannot be realized 
in which the RADAR sensor model provides the object list to the AEB function 
for the moving as well as stationary objects. The idealized sensor models have 
no occlusion effects, represented on the left hand of the figure 3.6. The cut-out 
delay time tco [s] and distance d{°[m] parameters of cut-out scenarios have no 
influence on present crash events, while the idealized sensor model constantly 
recognizes the object [obj1] as a relevant stationary object. 


— vte =30 km/h —A— v88°=35 km/h —8— v2 =40 km/h —y— v%8°=45 km/h 
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Figure 3.6: Safety envelopes based on comparison between simulation results of idealized sensor 
models (left) and high-fidelity sensor models (right) in the example of cut-out test 
scenarios. 


The left side of the figure 3.6 shows the three types of escalations (6!, &2 and 
&2) that occur at different Ego-vehicle velocities v% = {30, 35, 40, 45} [km/h], 
respectively, while a collision with the non-occluded object [objl] does not 
occur. Due to the margin scale of the figure 3.6, though it might seem that the 
Ego-vehicle collides with the object [obj1] after stopping, this is not the case. 
A positive distance between the Ego-vehicle and the object [obj1] remains even 
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after the Ego-vehicle stops due to the emergency braking ofthe AEB function. 
The same test scenarios are repeated with high-fidelity sensor models to show 
the influence of cut-out scenario parameterization with a crash event in the case 
of vs = 45 [km/h], as illustrated on the right side of figure 3.6. Meanwhile, 
an occluded object [obj] is simulated to less idealize the driving scenario, 
it is evident that the three types of escalations (6!, &2 and &3) happen at 
d’e!= {24, 20, 7} [m], respectively, resulting in a crash (with vy°° = 45 [km/h]) 
due to insufficient time to respond. In chapter 4, the developed HiL test platform 
is elaborately discussed with its implementation. 


3.3.3 Traffic Simulation-based Approach 


The introduction of automated vehicles will change the flow of road traffic. 
Therefore, a macroscopic statement about the safety of automated vehicles can 
be obtained by the traffic simulation based approach [PSS19]. The concept of 
traffic simulation is to simulate not only a single driving scenario but a whole 
road network with numerous road users (so-called Agents). Kitajima et al. 
[K*19a] developed a multi-agent simulation for estimating the impact of au- 
tomated vehicles on road traffic. Rösener et al. [R*18b] investigate the change 
in the occurrence frequency of scenarios to assess the safety performance 
of ADFs. Saraoglu et al. [SMJ19] present a framework called MOdel-Based 
Autonomous Traffic Simulation Framework (MOBATSim) for the analysis of 
traffic safety including automated vehicles with a focus on the fault-error- 
failure chain. In traffic simulation, the entire ODD can be simulated to increase 
the efficiency of the staged introduction of automated vehicles [Hal20]. Bach 
et al. introduce a model-based specification of driving scenarios with the 
example use case of an ACC system based on the abstraction of temporal and 
spatial information [BOS16, BHOS17]. Otten et al. extend the model-based 
scenario specification with an automated assessment and evaluation concept 
for stochastic digital test drives [OBW* 18]. 


To speed up the required FOT of the automated vehicles in car-following 
scenarios, Zhao et al. propose an accelerated evaluation method using stochas- 
tic optimization and importance sampling methods [Zhal6]. Although the 
simulation-based falsification is relatively similar to testing, a search is con- 
ducted within a logical scenario for a parameter set, where the ADF violates 
the requirements [AGZ18]. Koren et al. [K* 18b] use a reinforcement learning 
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formulation to identify the trajectories that are likely to be critical or lead to 
collisions. Thus, the Adaptive Stress Testing (AST) framework searches for the 
most likely path to a failure event [CDDCm19, L*15, L*18b]. Gangopadhyay 
et al. [GKD*19] use a Bayesian optimization to learn parameter values by 
observing the system’s output. Nabhan et al. [NSTH19] used a Random Forest 
model to detect the maximum amount of faulty scenarios in the search space. 


The falsification concept can be demonstrated using a driving scenario with 
crossing pedestrians for the verification of a multi-sensor fusion unit utilizing 
the implemented HiL test platform; Chapter 4 discusses such a concept in detail. 
The driving scenario distinguishes between the own lane and the adjacent lane 
with two classification areas. The pedestrian [obj1] crosses from the path of the 
moving Ego-vehicle to the adjacent lane and another pedestrian [obj2] comes 
from the opposite direction at the same velocity v,°° = yori = yor? = 5[km/h], 
as seen in figure 3.7. 


vel = 5 km/h 


Ego-vehicle 


Figure 3.7: Schematic diagram of a pedestrian crossing scenario with simulation-based falsifica- 
tion [ESW* 16]. 


The falsification is implemented using the Lemniscate of Bernoulli’s equation 
in the track co-ordinate system to obtain the lying-eight as a noise offset to 
the camera sensor model for object detection. The parametric equations for the 
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Lemniscate with a half-width [m] are ae! [m] and ar [m] ofthe camera-based 
detection, as shown in equation 3.1. 


cos(t) 


cos(t) * sin(t) 
sin?(t) + 1 


grel = 
( sin? (t) + 1 


r je ds ( yan (3.1) 


Figure 3.8 depicts the object detection from the RADAR sensor model (Seyi 
and Sie) and the camera sensor model Be , and a. The sensor models 
assign the track IDs based on the criticality of the two pedestrians [obj1] and 


[obj2]. While their walking paths cross each other, their track IDs are swapped. 
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Figure 3.8: Falsification of object detection (right) for verification of multi-sensor fusion using 
the example of a pedestrian crossing scenario compared to object detection without 
falsification (left) [ESS* 16]. 


Thereby, the Lemniscate noise model is applied to the object motion from 
the camera sensor model, so that the covariance matrix of the Mahalanobis 
distance can be identified. The covariance matrix represents the uncertainty of 
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the state estimation of longitudinal and lateral information. The Mahalanobis 
distance of the object is manipulated using the Lemniscate noise model until 
the fusion algorithm no longer associates the tracked object. Therefore, the 
covariance matrix can be determined over the entire ellipse. 


3.3.4 Scenario-based Approach 


In Japan, Antona-Makoshi et al. [J*19] explains the impact of the scenario- 
based approach on the safety assurance process for autonomous vehicles. 
Krishna [Kril9] extracts the corner case scenarios that occur on the roads 
of Singapore for automated vehicles. The corner-case scenario is one in which 
two or more parameter values are each within the capabilities of the system, 
but together constitute a rare condition that challenges its capabilities [fS*20], 
[Pon21]. In the Netherlands, the StreetWise methodology provides scenario- 
based safety validation of automated vehicles using a database of real-world 
scenarios [EPG* 18]. 


In Germany, the PEGASUS project relies on scenario-based testing to derive 
scenarios from system knowledge, domain modeling and field observation 
[WLFM19, PEG19]. The scenarios are applicable as test cases, which are 
executed and evaluated in the simulation or on test tracks [DG18]. The corre- 
sponding criticality metrics are employed to determine the automated driving 
capabilities [BKB*19, Hau21]. Schuldt [Sch17b] proposes the use of equiva- 
lence classes, boundary value analysis and combinational methods for identi- 
fying the representative driving scenarios. The proposed approach of Schuldt 
provides a systematic generation of test cases, but lacks a method to determine 
a meaningful test coverage [S*18a]. Schuldt motivates a scenario-based test 
process and presents a systematic test case generation by use of a four layer 
abstraction model. The concept of criticality analysis can be explained using 
the example of an off-tracking driving scenario. 


Figure 3.9 illustrates the rear axle path prediction of the off-tracking phe- 
nomenon based on Tractrix motion [ESS*19b]. The off-tracking phenomenon 
occurs when a vehicle turns and its rear wheels do not follow the same path as 
its front wheels [RA12]. 


The low-speed transient off-tracking describes the lateral offset between the 
turning paths of the front and rear axle before steady-state off-tracking is 
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reached. While commercial vehicles traverse shorter curves or curves of smaller 
radius, non-steady-state off-tracking increases gradually at low-speed vehicle 
turning scenarios. Thereby, the swept path width is the difference in paths 
between the outside front tractor tire and the inside rear trailer tire. 


Definition 3.6 (Tractrix Motion): The vehicle off-tracking behavior at low 
speeds is approximated by a Tractrix motion, where the rear axle of a tractor- 
semitrailer combination truck follows a given steering curve for low speed 
turning scenarios. The rear axle is always moving in the direction of the front 
axle based on a given wheelbase dw[m] [RA12]. The velocity of the rear axle 
depends on the direction of the velocity vector of the front axle. 


Definition 3.7 (TTC): It refers to the time required for two objects to collide; if 
they are moving at their current velocity and following the same path [W* 16]. 


Figure 3.9: Tractrix motion for low-speed transient off-tracking within a cornering scenario 
[ESS*19b]. 


Figure 3.10 shows the criticality analysis with a stationary pedestrian at 
different positions (pl, p2, p3, p4 and p5) and velocities of v°° [km/h]. The 
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off-tracking scenario is executed on the HiL test bench and managed by a test 
script for the right-hand turns with the SGA function. 
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Figure 3.10: TTC criticality analysis with a stationary pedestrian in the Tractrix zone using the 
SGA function within a cornering scenario [ESS*19b]. 
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In the automotive industry, HiL-based test methods are commonly used and 
they provide a significant advantage for the functional verification of software 
components on the ECU in the laboratory [Diis10]. The integration capability 
of perception sensors in HiL test benches reduces the dependence on simu- 
lation models and their validation in contrast to those in SiL environments. 
Accordingly, the Device under Test (DuT) can be evaluated at the hardware 
level and also its influence on the system, such as poor performance due 
to limited computing power or memory space [FBS09]. In addition, various 
effects, such as message loss, time delays and limited signal value ranges 
due to communication between hardware components, can be investigated. 
There are three main architecture levels of the HiL test system (component, 
subsystem and system). At the component level, each individual ECU can 
be verified with respect to its functional correctness. While ADFs commonly 
include several ECUs, the data flow can be verified at the subsystem or system 
level. Although the integration of diverse ECUs in a virtual test environment 
increases realism in simulation, the test system complexity can be unman- 
ageable [NGB13]. Consequently, the design of the HiL test system needs to 
meet the requirements derived from the desired test objective and not for every 
degree of realism or fidelity [Tell4]. 


4.1 Simulation Co-ordinate Systems 


The entire simulation relies on the right-handed inertial co-ordinate system 
R 5, where the x indicates the east direction, the y the north direction and the z 
the elevation direction. The rotation matrix RI is generated from the intrinsic 
Euler angles by multiplying the three matrices generated by rotations around 
the axes. 
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According to the ISO 8855:2011 for right-handed vehicle co-ordination 
systems [fS*11c], the rotation matrix RL follows a z-, y- and x- rotation 
sequence, as depicted in equation 4.1. 


cosy -siny O}}cos@ 0 sing} ıl 0 0 
RL = |siny cosy 0 0 1 0 |{0 cos@ -sind (4.1) 
0 0 1} |-sind 0 cos@| |0 sind cos® 


4.1.1 Ego-vehicle Co-ordinate System 


The origin of the Ego-vehicle co-ordinate system Rs in neutral loading 
conditions is on road level at the center of the truck rear axle. The Ego-vehicle 
co-ordinate system determines the position of environmental perception sen- 
sors and detected objects. The rotation matrix RẸ is an identity matrix, where 
the sensor co-ordinate orientation corresponds to the Ego-vehicle orientation, 
as illustrated in equation 4.2. The origin of the RADAR sensor co-ordinate 
system Rs is at the installation position of the RADAR, mounted on the front 
of the Ego-vehicle. 


0 0 
1 0 (4.2) 
0 1 


The rotation matrix RS describes the rotation from the inertial co-ordinates to 
the RADAR sensor co-ordinates, as shown in equation 4.3. 


R = (RERE) | (4.3) 


The transformation matrix T is calculated from the inversion matrix of the 
sub-transformations from Ego-vehicle into inertial co-ordinate systems a 
and from sensor into Ego-vehicle co-ordinate systems tT. as represented in 
equation 4.4. The distance vector ti, specifies the distance between the origin 
points of Ego-vehicle and inertial co-ordinate systems, while the distance vector 
t describes the distance between origin points of sensor and Ego-vehicle co- 
ordinate systems. 
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4.1.2 Object Co-ordinate System 


The origin of the object co-ordinate system Ro corresponds to the origin point 
of the detected object in front. Equation 4.5 represents the transformation 
matrix T, from the object to inertial co-ordinate systems, where the distance 
vector t describes the distance between origin points of object and inertial 
co-ordinate systems. 


R! t 
TL=| 0 0 4.5 
oT i (4.5) 


The origin of the detection co-ordinate system Rp allocates on road level 
at the center of the rear bumper of the detected preceding object. Figure 4.1 
depicts the various simulation reference co-ordinate systems of environmental 
perception sensors [Amm15]. 


[d,cils,[Vrels, [ares [dal o 


Figure 4.1: Simulation reference co-ordinate systems of environmental perception sensors in the 
example of detection of a vehicle ahead. 
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The equation 4.6 describes the Ego-vehicle velocity and acceleration vector 
components at the sensor co-ordinate system Rs as follows: 


ego ass? 

[vee], = [oye] = RP. [v], [a ]s = [as] = RP. [a]; (4.6) 
ego ace? 
Z S zZ S 


The equations 4.7, 4.8 and 4.9 depict the relative vector movement components 
at the sensor co-ordinate system Rs as follows: 


rel 
X 
l S pl l 
ee en 25). lo | (4.7) 
d= 
S 
rel 
x . 
[vt], = foe] = (BERS). [oe], - [els (4.8) 
viel 
z Is 
an 
[a]; = |a| = (RERS)- [2], - [als (4.9) 
a 
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The relative longitudinal movement components are represented by d'°' [m], 
v"! [m/s] and at! [m/s?], while the relative lateral movement components 
are defined by dj"'[m], uy" [m/ s] and a [m/ s?]. Also, the variables d!°' [m], 
vt! [m/s] and a'®![m/s?] represent the relative vertical movement components. 
The relative start reference co-ordinate system Rg is a special co-ordinate 
system to allow the zero initialization of the external vehicle dynamics simu- 
lation at its own origin points, when the Ego-vehicle needs to be initialized at 
a non-zero location or orientation within the driving scenario. In addition, the 
track co-ordinate system Ry is used to allocate and control objects through the 
reactive test automation. The co-ordinate axes are defined on the road, which 


are applied along the reference road center. 
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Equation 4.10 depicts the curvature of the driven distance of the Ego-vehicle 
in the sensor co-ordinate system. 


Kego = aa (4.10) 
Ux 


In case of the simulation of an eHorizon sensor via an OpenDRIVE[ASA21] 
database, the geographic co-ordinate system should be applied to the geo- 
graphical data based on World Geodetic System 1984 (WGS84) co-ordinates. 
The OpenDRIVE provides an open file format for the logical description 
of road networks[ASA21]. The camera co-ordinate system Rç represents the 
co-ordinate axes of the physical camera ECU. If the camera sensor is stimulated 
via a monitor, the monitor co-ordinate system R m is used, which represents a 
2D co-ordinate system. The origin of the monitor is located in the lower left 
corner [Nen14]. The co-ordinates of the monitor are normalized and valid in a 
range between 0.0 and 1.0. 


4.2 Vehicle Dynamics Simulation 


The dynamic behavior of heavy-duty trucks differs considerably from that 
of passenger cars due to the geometry and dimensions of the truck-trailer 
combinations. Also, the loading conditions have a considerable influence on 
the longitudinal and vertical positions of the vehicle center of gravity. The 
wheel configuration variants influence the off-tracking behavior of the trailer in 
cornering situations with different wheelbases, drawbars and kick angles. Pitch 
and roll movements need to be taken into account, which are strongly influenced 
by the aforementioned factors. Therefore, vehicle dynamics is an important 
topic to discuss. Moreover, a CMV consists of a driver’s cabin suspended from 
the vehicle frame in order to reduce mechanical road excitation. As a result, 
the environmental perception sensors have to cope with static and dynamic 
tolerances of truck-trailer combinations. The static tolerances relate mainly to 
sensor mounting position, vehicle type, tire pressure, type of cab suspension 
and load condition [SMAN08]. Additionally, the dynamic tolerances refer to 
the tractor cab movements. The vertical movement shifts the sensor height in 
relation to the road surface. The cabin pitch and roll angles change the optical 
axis position of the forward-facing camera mounted behind the windshield in 
relation to the horizontal axis. 
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Moreover, the steady-state behavior of a truck is determined in accordance 
with ISO 14792:2011 by the steady-state circular tests [fS*11b]. The change in 
the required steering as a function of lateral acceleration is represented by the 
under-steer gradient, which shall be positive for heavy vehicles. The negative 
under-steer gradient leads to instability at a certain critical velocity [MP17]. 


4.2.1 Truck Cabin Simulation 


Figure 4.2 shows the pitch movements when braking in straight ahead direction 
using a parameterized simulation model of the Mercedes-Benz Actros [Tru18] 
tractor with 4x2 axle configuration and a weight of 18 tonnes [Mar04]. The 
truck’s braking and driving dynamics differ from those of a passenger car. The 
pneumatic brake system has a time delay and slower response than the hydraulic 
brake system in passenger cars. Therefore, the load condition influences both 
dynamic stability and braking dynamics, whereby the difference between full 
and empty weight is very large. 


P, = 50% P, = 100% 


20 T T T 


Figure 4.2: Step-shaped half and full braking in a straight line with Mercedes-Benz Actros 4x2 
tractor simulation model at a constant longitudinal velocity of 80 [km/h] including 
braking torques (left) and pitch angles (right). 
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Moreover, the process of stopping a CMV in an emergency requires a complex 
interaction between the braking system, the CMV tires, the CMV’s dimen- 
sions, the loading conditions and the road surface characteristics. The realistic 
simulation of CMV dynamics enables the virtual movements of the envi- 
ronmental perception sensors to investigate the effects of cabin dynamics in 
braking situations. Therefore, the multi-body vehicle dynamics need to sim- 
ulate the roll and pitch movements of the truck’s tractor cabin. Meanwhile, 
the pitching movements during braking affect the detection performance of 
the perception sensors. If the brake pedals are set in a step-wise manner to Pp 
= 50% and 100%, braking torques are applied at half and full braking with 
Mp = 6.3 [kNm] and 16.5 [kNm] respectively. The pitch angles of the truck’s 
tractor cabin are shifted from 6, ~ 2.2° to 3.2° with an initial offset of 1° at a 
constant longitudinal velocity v,°° = 80 [km/h]. Figure 4.3 represents the roll 
movements when driving around a tight curve using the same parameterization 
of the employed simulation model. 
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Figure 4.3: Driving around a tight curve at lateral acceleration of 6 [m/s?] and longitudinal 
velocity of 80 [km/h] including roll movement compared to weight force on the rear 
axle (left) and Ego-vehicle trajectory (right). 
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The roll-over angle of the truck’s tractor cabin can be estimated on the basis of 
the tight curve maneuvers according to ISO 16333:2011 [fS*11a]. The calcu- 
lation of the tire lift-off and roll-over limits specifies the stability limit of the 
vehicle when the rear axle tires are lifted off the road. Thus, ISO 16333:2011 
presents a test method for estimating the maximum lateral acceleration that a 
CMV could withstand in steady-state turning maneuvers without rolling over 
[fS*11a]. From the right side of the figure 4.3, the Ego-vehicle accelerates in 
a tight curve maneuver, which is expressed in the left side of the figure 4.3 in 
terms of a corresponding roll angle of 6.53° with a weight of 0 [kN], which 
means that the rear axle of the Ego-vehicle does not exert any force. The roll 
movement of the truck’s tractor cabin is shifted to ¢2"** = 6.53° at actual weight 
on rear axle with F, = 0 [kN], lateral acceleration with a = 6 [m/s?] and 
longitudinal velocity with v,°° = 80 [km/h]. 


4.2.2 Run-time Analysis of Vehicle Dynamics 


The applied vehicle dynamics simulation uses an explicit Euler integration 
method based on Simulink® with a sampling rate of 1000[Hz]. While the 
turn-around time represents the computing time required to calculate the ex- 
ecuted tasks, the sampling time indicates the time interval for the integration 
step of the simulation. Therefore, the turn-around time should be shorter than 
the sampling time to ensure that the simulation is executed in real-time. The 
explicit Euler integration method utilizes the equations 4.11 and 4.12 for each 
Simulink® block at each simulation time step t= [ms]. The [mdlOutputs] 
task comprises the output part and the [mdlUpdate] task contains the update 
part of the discrete state-space equation. The [mdlUpdate] task updates the 
block inputs u, with the discrete state xq; for each Simulink® block that has a 
discrete state xg, while the [mdlOutputs] task calculates the block outputs yo 


Yo = mdlOutputs(ug, Xa, ts) (4.11) 
Xa+1 = mdlUpdate(ug, xq, ts) (4.12) 
As distinct from a general-purpose operating system, a Real Time Operating 
System (RTOS) is expected to meet computational time constraints when exe- 


cuting software applications. The implemented HiL simulator uses a 2.1 GHz 
Intel Quad-Core Ivy-Bridge processor with a VxWorks® RTOS. 
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Definition 4.1 (VxWorks® ): An embedded RTOS developed as proprietary 
software by Wind River Systems. The VxWorks® kernel utilizes a shared mem- 
ory model to control the communication between task states based on preemp- 
tive scheduling, where a first-in-first-out policy is used to schedule all tasks. 
The binary semaphores of VxWorks® provide a synchronization between the 
tasks controlled by the functions [semTake] or [semGive]. The semaphore is a 
binary variable for multi-process access control of acommon resource within a 
RTOS, where a binary semaphore acts as a flag that can be blocked or released 
[HVJ14]. 


The test cases are graphically modeled with special state machines on the host 
computer and transformed as byte code to the virtual machine task [JavaVM]. 
The compiled test cases thus run on the real-time HiL simulator using the 
[JavaVM] task and reactively control the synthetic driving scenario. Figure 4.4 
shows the turn-around time t,[s] of each task at the HiL simulator, where the 
total turn-around time does not exceed the sampling time ts = 0.001[s]. 


Median + pp —— 25% — 75% ——9% — 91% 


yk 
> 


RTOS Vxworks tasks 
% 
Ry 


$ 0 0.2 0.4 0.6 0.8 1.0 
to [ms] 


Figure 4.4: Turn-around time of task of vehicle dynamics, hardware in-/output and test automation 
under hard real-time conditions. 
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In this way, the [HWIO] task sets up the communication between the HiL 
simulation and the DuT. The entire execution time at the HiL is represented 
by the turn-around time and can be summarized in terms of [mdlOutputs], 
[mdlUpdate], [JavaVM] and [HWIO] tasks. The highest processing usage is 
provided by the [JavaVM] task, and then by the [HWIO] task, as illustrated in 
figure 4.4. For this reason, an optimization is required to use the available time 
and hardware resources better for further necessary simulation components. 
Consequently, the employed HiL test bench utilizes a distributed heteroge- 
neous co-simulation environment in order to realize a real-time capable system 
architecture. The co-simulation environment provides the core functionality of 
task and data management for the simulation of road traffic as well as Ego- 
vehicle driver and environmental perception sensors at a different sampling 
time. 


4.3 Road Traffic Simulation 


The fidelity of road traffic simulations can be divided into four categories: 
Macroscopic, mesoscopic, microscopic and nanoscopic. First, macroscopic 
simulations characterize the traffic flow, velocity and density of traffic (e.g. the 
number of road users who travel a certain distance per unit of time). Therefore, 
macroscopic fidelity is well suited for the analysis of large or complex road 
networks. Second, traffic units can be grouped in the mesoscopic models, where 
each group is treated as a single traffic unit (e.g. queues of vehicles). Third, the 
microscopic models simulate the behavior and interactions of each simulated 
traffic unit individually with specific state variables such as position, velocity 
and acceleration. The rules of vehicle behavior such as speed and lane changes 
are also taken into account. Fourth, an additional level of detail in nanoscopic 
models is achieved by dividing each vehicle into a number of sub-units. The 
nanoscopic model thus allows an extension of the vehicle dynamics modeling 
and the driver behavior fidelity. In this way, vehicle dynamics, complex driver 
decision-making processes or interaction with the vehicle environment can be 
modeled in more detail. In general, the computing time for traffic simulation 
increases considerably with increasing fidelity. Consequently, the HiL test 
system requires a compromise between fidelity level and computing effort for 
real-time traffic simulation under the given real-time conditions. 
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4.3.1 Multi-rate Co-simulation Setting 


The developed real-time co-simulation platform is utilized to evaluate the ADFs 
in a closed-loop HiL configuration. The Functional Mock-up Interface (FMI) 
co-simulation manages the scheduling and data exchange between multiple 
simulation units with independent solvers using VxWorks® callback routines 
[Sch16b], [Sch17]. As a result, the FMI integrates the heterogeneous modules 
to make one real-time-capable application for the HiL simulation. Figure 4.5 
illustrates the co-simulation integration scheme to provide the Ego-vehicle 
dynamics simulation with contact point information at a sampling rate of 
1000[Hz]. In order to place the Ego-vehicle correctly in the 3D space, the 
computation of the road contact is necessary. 


3D simulation environment VxWork®-based HiL simulator 
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Figure 4.5: Sequence and schematic diagrams of the road traffic co-simulation with exchange of 
contact point information using asynchronous real-time simulation. 
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The number of contact points between the tires of the truck-trailer combination 
and the road surface is 10 points (one contact point per tire) as the maximum 
number of contact points for the applied vehicle dynamics model, described 
in section 4.2. The real-time vehicle dynamics simulation needs contact point 
information at a higher frequency fy = 1000[Hz] than the frequency of the sim- 
ulation task management fm = 120[Hz]. Real-time operation in asynchronous 
mode utilizes a common time domain between simulation components to run 
them in real-time. The gradients are computed by each simulation component 
to handle and align asynchronously computed results. The data packages of 
tire-road contact points introduce the road profile in correspondence of the tire 
contact points. The task of Ego-vehicle dynamics interacts with the task of tire- 
road contact points to provide the tire longitudinal and vertical positions Xw, 
Yw respectively. The vertical displacement Zw of all road-tire contact points 
is determined and sent back to the task of Ego-vehicle dynamics, especially in 
uphill/downhill driving situations. The vector of road’s longitudinal gradient 
an, lateral slope œ!" and friction coefficient u, are assumed with constant 
values. 


4.3.2 Integration of HiL Simulation Modules 


The data management through equivalent FMI implementation provides the 
co-simulation scheme with high portability between various HiL simulation 
platforms and enables the run-time fault injection scheme triggered by the test 
execution, where as, the FMI standardization of interfaces are set in table 4.1. 
The road traffic simulation supports the traffic scenarios with visual and logical 
databases stored in an eXtensible Markup Language (XML) format [vNC14]. 
The scenario database is based on de-facto standard formats OpenDRIVE 
[ASA21] and OpenSCENARIO [ASA22] for specifying road networks and 
dynamic contents respectively [VNCDW09]. While OpenDRIVE road net- 
works assign static features to the driving scenario (e.g. lane, traffic sign, 
intersection, junction, crossing, etc.) [ASA21], OpenSCENARIO databases 
provide the synthetic scenario with the detailed dynamic contents of objects 
(e.g. actor, object, trigger, action, etc.) [ASA22]. Due to performance reasons, 
the change between anon-animated and an animated pedestrian depends on the 
distance between the Ego-vehicle position and the pedestrian position. If the 
distance to the pedestrian is less than a certain minimum distance, the animated 
pedestrian is used. 
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FMI category Description 
Actor vehicle: 
e vehicle category (e.g. car, truck, motorcyclist and bicyclist) 
e position (x[m], y[m], z[m]) and orientation (¢[°], @[°], w[°]) 
e driver behaviors (e.g. distraction and drowsiness) 
e vehicle behaviors (e.g. overtaking and rear-ending) 
e wheel information (e.g. steering angle, radius and forces) 
e power-train information (e.g. speed and torque) 
pedestrian: 
e pedestrian category (e.g. single and group) 
e position (x[m], y[m], z[m]) and orientation (¢[°], @[°], w[°]) 
e pedestrian behaviors (e.g. crossing, inattentiveness and failure 
to obey traffic laws) 
e gesture and motion pattern (e.g. run and walk) 
Road traffic road information: 
e road geometry (e.g. curvature information, intersections, 
roundabouts and rural areas) 
e road condition (e.g. road damage, uneven surfaces and road 
construction) 
e road type (e.g. urban, rural and highway) 
e traffic condition (e.g. traffic sign, traffic light, high speed limit, 
heavy flows and potential accidents) 
e lane marking (e.g. type, color, broken/missing markings and 
irregular lane/road shapes) 
e drive lane information (e.g. width and ID) 
e static objects (e.g. fence, pole, vegetation and curbstone) 
Camera ECU image generator: 
e image height and width 
e depth and camera information 
ambient conditions: 
e time of day, sky state and visiblity 
e illumination (e.g. shadow, night, dawn/dusk and directly facing 
the sun) 
e road conditions (e.g. dry and wet) 
e weather conditions (e.g. fog, rain and snow) 
RADAR ECU sensor stimulation: 
e Over-The-Air movable antennas for the horizontal positions 
e distance, velocity, and size of the RADAR objects 
e radial distance, Doppler velocity, azimuth angle and elevation 
angle of the RADAR detections. 
Virtual sensor sensor simulation: 
e data communication (e.g. detection, feature and object level) 
e sensor position and orientation 
e relative movement of recognized objects (d‘*'[m], ae [m], vt! [m/s], 
vi! [m/s], af" [m/s], af! m/s?) 


Table 4.1: List of information categories of data management with equivalent FMI modules via 
well-defined in-/output interfaces. 
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The developed real-time co-simulation platform is utilized to evaluate the 
automated driving ECUs in a HiL configuration, as demonstrated in figure 4.6. 
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VxWork®-based HiL simulator with DuT and Camera-box 


Figure 4.6: Schematic diagram of the dynamic behavior testability of ADFs within a HiL frame- 
work. Partially mentioned in chapter 3, and is integrated within chapter 4. 
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The multi-rate numerical simulation enables the integration of heterogeneous 
simulation modules (sensor models, vehicle dynamics and test automation) 
using the User Datagram Protocol (UDP) socket interface mechanism. The 
real-time HiL simulator uses two Ethernet connectors to access both the sim- 
ulation environment via Transmission Control Protocol (TCP)/UDP ports and 
the host computer via TCP ports. In addition, the monitor camera box is con- 
nected to the HiL simulator for the Over-The-Air (OTA) stimulation of the 
camera ECU. The monitor is connected to the simulation environment via a 
Digital Visual Interface (DVI) connector [Elg12]. 


The ECU in-/outputs can be plugged into the break out box for measurement and 
hardware fault injection purposes. The Automotive Data and Time-triggered 
Framework (ADTF) tool provides a measurement framework for image recog- 
nition algorithms based on received data. The test automation program manages 
the actions of all actors, such as lane changes or speed changes by the traffic 
module, to control the Ego-vehicle and other traffic objects. The target accel- 
eration and steering are translated with the vehicle dynamics simulation into 
throttle, brake and steering wheel positions within the applicable limits. 


The model-based test methods support reactive tests, where the execution of 
a test case depends on what the DuT is doing while being tested [ESW* 16]. 
Therefore, the model-based testing is integrated into the closed-loop HiL plat- 
form, as illustrated in figure 4.6. Furthermore, the robustness tests are integrated 
by run-time fault injection to assess the degree of correct functionality under 
invalid inputs or in stressful environmental conditions. For the minimization 
of human effort in test execution, test automation provides computer-aided 
execution of dynamic verification of a function’s behavior on a limited number 
of test cases against the specified expected behavior. 


4.3.3 Round-Trip Time Estimation 


The delta time of each simulation step is identical (fixed-step solver) but the 
real-world time between the steps differs and, therefore, influences the correla- 
tion between simulation time and real-world time. A system that accumulates 
simulation time in-sync with the progress of real-world time is called a real- 
time system. 
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There is a distinction between hard and soft real-time systems based on the 
consequences in case of a violation of the real-time requirements. While the 
hard real-time systems must reliably deliver the correct result within the 
required response time, the soft real-time systems can statistically meet the 
required response time within a tolerance margin [LSE12]. Consequently, 
the co-simulation framework allows the execution of heterogeneous real-time 
simulation components, whereby the real-time data exchange must not exceed 
the defined tolerance margin. The real-time requirements have to be fulfilled 
for the HiL test bench for ADFs. Hence, the real-time behavior validation of 
the distributed HiL components is typically as important as their functional 
correctness. Accordingly, a case study is conducted to investigate the time 
constraints using a multi-sensor data fusion module, a camera sensor model Se 
and aRADAR sensor model Sr, as demonstrated in figure 4.7. 
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Figure 4.7: Driving scenario setup with approaching a stationary object for validation of the timing 
constraints within the HiL co-simulation framework. 
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The camera sensor model calculates the relative longitudinal distance d$ be- 
tween the Ego-vehicle and the stationary object. Similarly, the RADAR sen- 
sor model computes the relative longitudinal distance d\ between these two 
vehicles. The sent timestamp message 6! is employed for the fusion of non- 
synchronized measurements from heterogeneous sensor models using received 
timestamp messages 0° and ©". 


The above-mentioned case study applies a driving scenario in which the Ego- 
vehicle approaches a stationary object. The driving scenario integrates an AEB 
function with reaction to the stationary object within the HiL co-simulation 
framework. Consequently, approaching a stationary object is carried out at 
different velocities v\°° = {5, 10,---, 105} [km/h] to evaluate the time con- 
straints. If the Ego-vehicle velocity is increased, the temporal longitudinal 
distance to the stationary object is shortened. The number of corresponding 
timestamps decreases accordingly. In order to determine the Round Trip Time 
(RTT) at the soft real-time communication, the difference between d$ [m] and 
dł [m] is calculated. The RTT can be calculated at the start time of the same 
sent and received timestamp messages, as described in equation 4.13. Thus, 
the RTT describes the time interval to send a message from a starting point to 
a destination and return to the same starting point. 

dy 


RTT = ~~~ y Ô -Ô =ô" (4.13) 
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we 


N! 
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The co-simulation interface introduces latency that determines the limits of 
the real-time capability for the entire framework based on the configuration 
frequencies, where fy = 1000[Hz] and fm = 120[Hz]. Figure 4.8 presents the la- 
tency between timestamp measurements from the camera and RADAR sensor 
models without taking the additional delay, caused by the CAN bus interface, 
into account. Although the Ego-vehicle velocity increases, the RTT remains 
at fm = 120[Hz] with a variation between +2 simulation cycles with a reduced 
number of matching timestamps. This variation is considered as an accepted 
tolerance for the soft real-time conditions. 
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Figure 4.8: RTT calculation at start time of the same timestamp messages for sensor fusion using 
different velocities of the Ego-vehicle in the scenario with approach to a stationary 
object. 


4.4 Perception Sensor Simulation 


The HilL-based test approaches can provide an efficient functional evaluation 
of environmental perception sensors under reproducible conditions. There are 
different methods, which can be practiced at three logical interface levels 
(detection, feature and object level) [fS*21] to stimulate the sensor behavior 
in a HiL environment. Therefore, the quality of the environment simulation 
depends on the layer of injection into the DuT. While injecting the data via 
the physical sensor layer or the unprocessed raw data, the simulated data has 
to represent the real world at a high fidelity level [NS11]. 
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4.4 Perception Sensor Simulation 


4.4.1 Sensor-in-the-Loop Testing 


Diewald et al. presented an antenna coupling approach for an OTA RADAR 
sensor stimulation, which considers the three subsequent processing steps 
(measurement, processing and observation) [Die15]. The stimulation describes 
the act of manipulating an entity in which its state corresponds to a driving 
scenario. The antenna coupling simulator allows dynamic simulation of moving 
objects using opposite RADAR antennas. Gowdu et al. improved the anten- 
na coupling approach to stimulate automotive RADAR sensors in a virtual 
electromagnetic environment using an OTA interface with wideband horn an- 
tennas at 77 GHz [GA* 18a]. The employed RADAR target simulator generates 
back-scattered radar signals to emulate the DuT by synthetic RADAR target 
signatures with specific attributes like RCS, range and velocity. 


Weiskopf et al. described various approaches of digital RADAR signal injection- 
based integration for RADAR ECUs in a HiL setup [W*15]. Furthermore, 
Hanke et al. proposed a way to design a realistic description of the automotive 
RADAR system with the help of sophisticated sensor models, whereby the data 
is generated synthetically at a phenomenological level [H* 15, H* 12]. The same 
integration approaches can also be employed to verify the camera based functi- 
ons in a 3D synthetic testing environment. Nentwig and Stamminger worked on 
the applicability of real-time generated computer graphics for camera-based 
detection algorithms [NS11]. Tan and Hassan presented a projection-based 
approach which is used to stimulate the camera ECU using synthetic traffic 
scenes [TH13]. The 3D scenes are visualized in real-time on a flat monitor. The 
3D orientation of the camera is estimated within a global frame of reference, 
whereby the video is displayed in a 3D virtual environment [HH15]. 


The 3D orientation is converted into the Camera Reference Frame (CRF), 
which is compared to the FOV of the camera ECU [SS13]. In order to match 
each monitor pixel with a point in the CRF, the camera ECU calibration with 
respect to the monitor is to be performed [NMS12]. In particular, the camera 
ECU is characterized in terms of its intrinsic parameters, e.g. focal length, 
skew coefficient and distortion coefficient [HM* 14]. The extrinsic parameters 
of the camera ECU with respect to the 2D screen are the rotation matrix 
and the translation vector [Kan16]. The screen co-ordinate system is a 2D 
system, where the x-axis corresponds to the right hand direction and the y-axis 
corresponds to the upward direction. 
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The camera ECU records the artificial scene and provides the image process- 
ing unit with the raw image data in a digital form. The image processing unit 
generates the object list from the resulting image data to the CAN vehicle 
bus interface and tracks the objects based on the simulated Ego-vehicle state 
data. Pfeffer and Haselhoff illustrated the injection-based approach, in which 
the synthetic raw image data is injected directly into the image processing 
unit via the bypass of the image sensor module [PH16]. The standardization 
of sensor injection interfaces play a crucial role, while the current stimuli 
injection interfaces are highly product-specific. 


Table 4.2 — terminologies are discussed in section 2.3 — summarizes the man- 
ifold ways of injecting synthetically generated stimuli into the DuT. Because 
of the technical limitations of covering all the sensor physical characteristics, 
the suitability of each approach should be determined according to the required 
test objective of the environmental perception sensor. 


Stimulus Over-The-Air Raw-data Target-list Object-list 
injection stimulation simulation simulation simulation 
DuT hardware not sensor not not 
modification required specific required required 
DuT software modification not software software not 
required bypass bypass required 
Additional hardware sensor hardware not not 
dependent adapter required required 
Simulation quality sensor high simplified simplified 
dependent 
Sensor ECU pipeline complete excluding excluding excluding 


measurement measurement measurement, 
& processing processing & 
observation 


Table 4.2: Overview of different Sensor-in-the-Loop approaches and their advantages and draw- 
backs [FHW 16, Feil8]. 


4.4.2 Object-list based Sensor Models 


The environmental perception sensor simulation is positioned on the Ego- 
vehicle and detects the objects within its pyramid-shaped cone. The sensor 
position is computed relative to the carrying CMV’s reference point (typically 
the center of the rear axle on ground level). 
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4.4 Perception Sensor Simulation 


The sensor uses a purely geometrical approach, which calculates the nearest 
point of extended object within the applied FOV. The modular architecture 
enables an iterative development of the phenomenological sensor model to 
achieve increased realism, as presented in figure 4.9. 


Q 
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H[P?] 


— t 
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Figure 4.9: Modular architecture of the phenomenological sensor simulation [ESFG19b]. 


Equations 4.14 and 4.15 describe the sensor model components. Therefore, 
the sensor model S™ maps a set of detected objects P™ from the Ego-vehicle’s 
environment to a set of tracked objects Ô™ into the object-level data fusion 
algorithm. The characteristics of the sensor mapping H are divided into H} 
modules with v € [1, Ny]. The set of configuration parameters for module H? 
is denoted by 6%}; and comprises the relevant subset of sensor properties. 


s™ <P" Q" (4.14) 
H =H, [6g, | 0--- oH? [63] oH? [pr] (4.15) 


Therefore, an incremental development process of simulation models is pro- 
posed to reproduce the physical relationship of reality with proper modeling 
depth. The procedure consists of the analysis step, the implementation step, 
the verification step, and the accreditation step. 
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The analysis step obtains aconceptual model from reality through analyzing the 
physical principle of the real component. The goal of this action is to construct 
specified instructions for the implementation of an executable model. The 
executable model is a computer program, which should be verified based on 
its specifications. The accreditation process compares the test results with its 
component test bench to determine whether the executable model’s accuracy 
is adequate to accomplish its intended function. The influence of sensor mod- 
eling is investigated on the simulation results using typical driving scenarios 
from the scenario database. 


The ADASIS interface is applied as a proper interface for information exchange 
to enable access to relevant map data via the CAN vehicle bus interface for 
advanced assistance features. Therefore, the eHorizon can be perceived as a 
virtual sensor to anticipate the driving path based on a non-physical measuring 
principle [BMBL06, KP*14]. The eHorizon is implemented as a sensor model 
with a predefined declaration of input information and defined output signals. 
The sensor model has an access to the map provided by the environment simu- 
lation and the position of the Ego-vehicle in this map. Using this information, 
the relevant section of the map is parsed and the position of the Ego-vehicle is 
obtained in path co-ordinates [HS15]. Along the aforementioned MPP, all the 
information is extracted and ADASIS v2.0 conform messages are generated. 
The configuration of the sensor model can be set up from the test automation, 
allowing the test cases to be linked to specific sensor configurations. 


The modular architecture enables iterative development of the eHorizon sensor 
model to achieve increased realism. The MPP consists of data structures that 
are provided through an OpenDRIVE parser [ASA21]. In reality, discrepancy 
between visual and map data is omnipresent due to map errors, old map data or 
optical detection failure. Through fault injection, defined inconsistencies can 
be produced within the HiL simulation environment. Amongst others, the fault 
injection covers the placement and value of traffic signs. These failures can be 
used for robustness testing of the fusion algorithms. The implementation of 
the eHorizon sensor model focuses on the basic road geometry and attributes 
(e.g. speed limits, overtaking signs) along the MPP, as shown in table 4.3. The 
ADASIS v2.0 is based on a path offset model, where the path is the first entity 
that must be obtained to retrieve the required information identified by a path 
identifier [ESAT17]. 
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Message Description 

META DATA semi-permanent data (e.g. country code) 
POSITION vehicle positioning in geographic co-ordinates 
SEGMENT road ahead data (e.g. speed limits, tunnel) 
STUB branch points (e.g. turn angle, intersection) 
PROFILE SHORT road’s course data (e.g. curvature, slope) 
PROFILE LONG road specific data (e.g. longitude, latitude) 


Table 4.3: List of eHorizon sensor messages and their descriptions [Are16]. 


The [META DATA] message contains country-specific information, where 
implicit speed limits are included implicitly for each relevant country. The 
[POSITION] message is determined by a path identifier and an offset along 
the path. The start of each path is defined to be at offset 0. The [SEGMENT] 
summarizes the most important attributes for a part of path. The [STUB] mes- 
sage defines the relationship between the paths. While the [PROFILE SHORT] 
message contains 10-bit variables for road’s course data, the [PROFILE LONG] 
message holds 32-bit variables for geographical co-ordinate information. The 
data of interest are either on the same path or on one of the sub-paths ahead of 
the Ego-vehicle. In order to fulfill the highly automated driving requirements, 
the ADASIS v3.0 is responsible for transferring high precision and up-to-date 
map data from the cloud to the ECU. Up-to-date information about traffic and 
road conditions are not provided by the ADASIS v2.0 protocol. 


The capability to integrate vehicle-independent information sources by ex- 
tending the environment simulation makes the proposed framework adaptable 
for further applications, such as Vehicle to X(everything) (V2X) communica- 
tion!. The eHorizon data contains vehicle position data as well as road segment 
attributes, such as road geometry, road class, number of lanes, speed limits, 
etc. The iterative development process enables the re-design of the modular 
simulation models based on the claimed modeling depth ranging from ideal to 
phenomenological sensor models. 


! The V2X communication summarizes various vehicle technologies that enable a vehicle to 
communicate, e.g. with other vehicles, with systems integrated into the infrastructure, with 
pedestrains’ mobile devices, or to a database. 
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4.5 Ontology-based Scenario Management 


The ontology-based NL notations are meta-data representations of the data 
elements and their semantic relationships in a structure that is understandable 
for humans and machines [Z*18a]. An ontology is equivalent to a Description 
Logic (DL) knowledge base [HPSVH03]. While knowledge representation in 
an ontology is based on DL, the Ontology Web Language (OWL) is acommon 
popular file format for storing ontologies based on the Resource Description 
Framework (RDF) data graphs [M*15]. Therefore, the ontology comprises 
Terminological Box (TBox) and Assertional Box (ABox) statements [EH16]. 
The TBox statements describe object-oriented classes within a knowledge base 
as a schema or data model. In contrast, the ABox statements are associated 
with instances of these classes [CK18]. 


Automated driving involves the use of ontologies in various applications, 
especially in situation assessment, scene understanding and behavioral planning 
[BMM18]. Armand et al. describes the application of ontologies to model 
interactions based on spatial-temporal relationships between road users and 
infrastructure [AIGZ17]. The sensor data is employed as ABoxs of an ontolo- 
gy to develop a human-like understanding of scenes. The scene understanding 
relies on object tracking, map data and the dynamic states of the subject vehicle. 
Behavior rules are stored in the semantic web rule language to infer knowledge 
from the TBoxs to a given ABox from the sensor data [KLN*18]. 


Ulbrich et al. proposes an environmental model derived from a knowledge base 
with hierarchical classes and relations between the entities [UNMH14]. The 
environmental model is updated by sensor data and utilized for online deci- 
sions. Geyer et al. proposes nomenclatures of a unified ontology for generating 
test cases and scenario catalogs [GBF*13]. However, Geyer et al. describes 
that each scenario catalog shall have its own nomenclature and concepts for 
knowledge organization. 


Figure 4.10 shows an ontology-based test scenario synthesis based on knowl- 
edge discovery from triggered FOT events. The systematic test case generation 
leads to concrete scenarios and test cases based on a generic model consisting 
of four layers of scenario description. The first layer belongs to road geometry, 
the second layer to static objects, the third layer to dynamic objects and finally 
the fourth layer to weather conditions. 
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The test cases are also deduced from the functional specification, which results 
from the top-level requirements and the use cases, whereby the use cases are 
also inferred from the top-level requirements. A category of adequate and 
relevant scenarios for existing field tests is extracted using the ontology-based 
scenario management. The semantic representation of worst-case scenarios 
can be obtained by using data mining techniques and systematically processed 
in requirements for ODD coverage. 
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Figure 4.10: Coverage-driven test concept with systematic test case generation based on field- 
based observation. 
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Definition 4.2 (Data Mining):It is an integrated Knowledge Discovery in 
Databases (KDD) process for the automatic identification of useful informa- 
tion in big data repositories [FPSS96]. Therefore, the data mining process 
comprises several transformation steps, from data pre-processing tasks to post- 
processing of the data mining results. The knowledge management process 
serves to obtain representative driving situations that are recorded by FOTs in 
the form of time-series of environmental perception sensor data. 


The implemented framework facilitates a functional verification of ADFs pre- 
cisely and more efficiently on the target ECU in the laboratory. The test cases 
are executed on a HiL co-simulation platform. The ontology combines relevant 
entities in natural language and assigns a formal order through conceptu- 
alization. Irrelevant entities or relations are excluded for the traceability of 
parameter changes associated with an ontology-based scenario synthesis. The 
ontology provides a method to derive possible observations of ABoxs from 
modeled knowledge of TBoxs [LTW20]. Subsequently, the ontology-based 
scenario synthesis can be converted into de facto simulation data formats (e.g. 
OpenDRIVE [ASA21], OpenSCENARIO [ASA22], etc.). 


The identification of situations from FOTs using data mining techniques offers 
a valuable solution for the generation of simulated test scenarios to extend the 
validity of test coverage of ADFs more cost-effectively [EUA*19]. Although 
the corner case scenarios occur rarely in real traffic, the ontology-based ap- 
proach provides the relevant parameter space from open-loop sensor signals 
to ensure the validity of the generated closed-loop test cases. The ontology- 
based transformation rules provide the syntheses of relevant driving scenarios 
based on the parameterized characteristic waveforms for the HiL scenarios. In 
case of further induced traffic situations, the classification maps them into one 
of the predefined categorical classes. Subsequently, the ADFs can be verified 
effectively and efficiently through an ODD coverage using test oracles based on 
envelope components of pass and fail criteria [L*19]. The proposed test con- 
cept complements the traditional knowledge-based text matrix and enriches 
the test cases with a maximum test coverage. Furthermore, the context-driven 
verification approach determines how such an argument can be formed by de- 
composing the test coverage and objectives. Therefore, the proposed approach 
is complemented by the systematic provision of various evidence of ODD 
coverage to meet the required test termination criteria. 
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Safety in automated truck driving can be maximized if human-like errors 
of automated CMVs can be avoided. Hence, the sense-plan-act robot control 
procedure is an essential part of the automated CMV’s response to the dynamic 
driving environment. Functions of the environmental perception and situation 
analysis are responsible for the recognition of the vehicle environment and the 
associated situational awareness. While the planning function is responsible 
for determining the driving trajectory, the motion control function operates 
the throttle or brakes, turns the wheels and otherwise actuates the vehicle to 
follow the plan precisely [Smil7]. To validate these functions, the field tests 
are carried out after verifying the absence of logical errors and determining 
the required total mileage and geographical variation [Ebn14]. Accordingly, 
the automated CMV fleet is equipped with data-logging devices for recording 
automotive communication buses, sensor raw data, additional context cameras 
and inertial measurement systems (e.g. Inertial Measurement Unit (IMU) and 
Global Navigation Satellite System (GNSS) sensors) for precise vehicle posi- 
tioning. Then, data mining techniques are usually used to retrieve novel and 
useful patterns from large databases [FPSS96]. 


5.1 Measurement Methods for Scenario Mining 


Scenario-based testing relies on the measurement data from real-world scenar- 
ios to derive the required knowledge within the scenario elicitation process. 
Consequently, the measurement method requirements include the acquisition of 
data with reasonable effort, while ensuring an adequate quality of the dynamic 
scenario description and the naturalistic behavior of the road users [Kril9]. 
Common measurement methods for obtaining data in scenario-based testing are 
vehicle data loggers, drones equipped with cameras and roadside infrastructure 
sensors. 
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Various publications identify the requirements of automated driving systems 
based on the German In-Depth Accident Study (GIDAS) accident database 
to prevent as many such human-like accidents as possible [S*19a, S*19b, 
FWP*19]. According to the National Motor Vehicle Crash Causation Survey 
(NMVCCS) database conducted by NHTSA for U.S. police-reported passen- 
ger vehicle crashes, driver-related contributing factors can be divided into five 
categories: First, the sensing and perception factors contribute with 24% to 
accidents caused by unrecognized hazards. Second, the incapacitation factors 
represent 10% of accidents due to drivers who are alcohol-impaired or other- 
wise incapacitated drivers. Third, the prediction factors due to a misjudgment 
of the other vehicle behavior account for 17%. Fourth, the factors of planning 
due to illegal maneuvers or poor decision making behind compliance with 
traffic rules and defensive driving cause 39% of the accidents. Fifth, the factors 
of execution and performance share with 23% due to inappropriate vehicle 
control. While a crash event can result in multiple common factors, the total 
percentage is more than 100% [M*20b]. 


5.1.1 In-vehicle Data Loggers 


The development of automated driving algorithms requires extensive tests 
on real-world driving scenarios, which are collected and aggregated from 
the FOT [L*18a]. The FOT represents a study undertaken to evaluate an 
ADF under normal operating conditions in road traffic environments [Mas 19]. 
Consequently, data loggers are installed in test fleet CMVs to record raw 
data from the perception sensors and data from vehicle networks in a time- 
synchronous way [AADN*16]. Thus, the data loggers for test fleets allow an 
in-depth analysis of the entire data flow from sensing, perception, prediction 
and planning to motion control software modules [ZWZ18, GKS18, KYB 18]. 
In addition, a number of public data-sets, such as the Karsruhe Institute of 
Technology and Toyota technological Institute data-set (KITTD [GLSU13], the 
Cityscapes data-set [COR* 16] and the truck-specific TuSimple data-set [N* 18], 
were published, which can be utilized for scenario-based testing. Section 5.2 
discusses in-vehicle data loggers describing their implementation. 
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5.1.2 Drones Equipped with Cameras 


Trajectories of individual road users can be extracted from an aerial perspective 
using drones equipped with high-resolution cameras. While the traffic is not 
affected by the measurement, computer vision algorithms are used to process 
the extracted naturalistic images [K*18c]. Several public data-sets provide 
vehicle trajectory data with the bounding-box based annotation of the detected 
vehicles. The highway Drone data-set (highD) provides a large-scale data-set 
of naturalistic trajectories of vehicles such as cars, trucks and buses on German 
highways using unmanned aerial vehicles [KBKE18]. The Stanford drone data- 
set consists trajectories of VRUs, such as pedestrians and bicyclists, extracted 
from drone video recordings of the university campus [R* 16]. The intersection 
Drone data-set (inD) contains trajectories of naturalistic road users at German 
urban intersections [BKM* 19]. The road user trajectory data-sets can be used 
for scenario-based testing of prediction and path-planning software modules. 


5.1.3 Roadside Infrastructure Sensors 


The infrastructure can be equipped with sensors installed on roadside masts 
to detect traffic or weather conditions. Ground truth bounding boxes can be 
extracted by permanently monitoring acertain road segment. If both vehicle and 
infrastructure data are collected separately, the two data-sets need to be time- 
synchronized with a GNSS clock for scenario reconstruction. The TAF-BW 
test field provides a distributed intelligent infrastructure that can handle traffic 
light states, road topology and data about monitored road users [FDW*18]. 
The test field in Lower Saxony covers sections of motorway with a variety of 
traffic environments and situations [R*15]. 


5.2 In-vehicle Data Logging System 
The Automated Driving Data Recorder (ADDR) gathers data from the ADFs 


and from sensors mounted on the truck. Figure 5.1 shows an in-vehicle data 
logging system and data analysis concept for the worldwide validation of ADFs. 
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In accordance with the circular buffer concept, representative driving situations 
in FOT at object list level can be separated from permanent recording in the 
form of time-series data from environmental perception sensors. The triggering 
events can be individually defined in the data logger depending on the DuT 
reaction, so that the event-based recordings are transported to the databases 
[BBLF19]. The recordings include vehicle bus data, sensor object lists and 
reference video streams for data analysis purposes. 


automated drivin, 5 
g corner case detection cloud-based data storage 
data recorder 
data logger data ingest station scenario database 
(scenario identification) (scenario extraction) (replay simulation & scenario fuzzing) 


Figure 5.1: Data handling structure of the in-vehicle data logging system with its main elements. 


The geographically distributed data sources constitute a challenge for existing 
development processes and information technology infrastructures. Therefore, 
a data ingest station is necessary for data storage and retrieval, where the corner 
cases can be identified for replay purposes. Also, the cloud-based data storage 
contains big data tools, architectures and analytics, that provide the database 
infrastructure for ingesting, storing, accessing and processing of logged data 
simulations [SPW*19]. For an objective assessment of the driving function 
used, suitable evaluation criteria are integrated into the database. 


5.2.1 Automated Driving Data Recorder 


According to the California Code of Regulations (CCRs) §228.06, ADDR 
is a mechanism installed in a CMV under test. This test CMV is equipped 
with an ADF to record technical information about the status and operation 
of the environmental perception sensors either continuously or for 30 seconds 
preceding a critical event (e.g. disengagement, crash, etc.) [LS19]. The data 
captured by the ADDR and stored in a read-only format shall be accessible and 
retrievable with a commercially available tool [DMV18]. According to SAE 
J3197, the ADDR does not interfere with the ability of the ADF to perform the 
DDT [Int20]. While the automated driving technology is still being developed 
and is not yet commercially deployed, the data included in the ADDR is used 
for validation and replay purposes. 
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Meanwhile, the SAE J1698 Event Data Recorder (EDR) is mainly used for 
traditional accident reconstruction analysis [Int17]. Since 2017, the German 
law requires data processing of disengagements with storage of the position 
and time data captured by a GPS antenna for vehicles with highly or fully ADFs 
[BMV 17]. Therefore, data has to be timestamped in a way that allows perfect 
synchronization of multiple data streams within the ADDR using a Precision 
Time Protocol (PTP) grand-master. According to the IEEE 1588-2019, PTP 
is a protocol used to synchronize clocks in a networked measurement system 
[IEC* 19]. Figure 5.2 illustrates an ADDR with real-time streaming of data 
source and sink components [BES*21]. 


real-time streaming of data source and sink components 


component data logger HMI and operating computer GPS antenna diagnostic computer 


(environment perception) | | (diagnosis of operating conditions) 


(PTP grand-master clock) (breakout boxes) 


i 


Ethernet switch with IEEE 1588 support 


I 


system data logger measurement trigger reference cameras data transmission 
(automated driving function) (manual/automatic/permanent) (contextual monitoring) (Ethernet or LTE) 


——> temporary connection 
——> logical communication 


Figure 5.2: Schematic diagram of an ADDR with its main components. 


The component data logger records the raw data of the environmental per- 
ception sensors, which can be regarded as a data source. For the component 
data logger, the Plug On Device (POD) interface allows access to the internal 
ECU software of environmental perception sensors as a standardized hardware 
interface [ASA17a]. Additionally, the system data logger collects the ADFs 
data as a data sink, which is communicated via an Ethernet switch capable of 
supporting IEEE 1588. The CMV driver utilizes the Human-Machine Interface 
(HMI) and operating computer to start, stop and monitor the measurements. 
Moreover, the HMI acquires the health status of the various measuring com- 
ponents via diagnostics interfaces, such as Universal (X) Measurement and 
Calibration Protocol (XCP) [ASA17b] and REpresentational State Transfer 
- Application Programming Interface (REST-API), to control the start/stop 
of the measurements. The ADDR provides contextual monitoring to gather 
information about the surrounding traffic. 
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Therefore, the reference cameras assist the analysis process to understand the 
underlying context in relation to the surroundings. For commissioning, the 
diagnostic computer is temporarily connected to check the prerequisites of the 
measuring components and ECUs via breakout boxes. The triggered measure- 
ments typically use a ring buffer to record data before and after the relevant 
event. The triggering events serve as a supplementary source of information to 
identify relevant driving scenarios occurring in the FOT [K*18a]. 


In practice, two types of recording have become established: Continuous and 
triggered measurements. If the recording is continuous, a reduced set is record- 
ed with the most important measurement signals for statistical statements in 
order to keep the data volume manageable. If certain situations are triggered, 
the entire vehicle bus can be recorded. For each event, a recorded video snip- 
pet of the traffic situation around the vehicle is generated, supplemented by 
information on system states and signal characteristics. The triggered measure- 
ments cover a shorter period of time, which is concentrated on certain driving 
situations and is automatically triggered when predefined conditions are met 
[dGP17]. In case of false negative events, a manual trigger button provides the 
trigger to record situations in which the system intervention is missing. 


As mentioned in section 2.5 to the ordinance implementing the act amending 
the road traffic act and the compulsory insurance act, the CMV with a SAE 
LA ADF needs to be equipped with a digital data storage system [Bun22], as 
follows: 


Ordinance (Event-based Data Storage): 

A data storage system must be integrated in the motor vehicle with autonomous 
driving function that collects, uses and stores data concerning the motor vehic- 
le with autonomous driving function on an event basis and during operation in 
accordance with §9(5) and $15 only for the purpose of improving road safety. 
The data to be collected is conclusively laid down in §1g(1) of the road traffic 
act in conjunction with Annex 2 to this ordinance." 

Im Kraftfahrzeug mit autonomer Fahrfunktion muss ein entsprechender Da- 
tenspeicher integriert sein, der ereignisbasiert und wahrend des Betriebs nach 
§Absatz 5 und §15 Daten des Kraftfahrzeugs mit autonomer Fahrfunktion 
ausschließlich zu dem Zweck der Verbesserung der Verkehrssicherheit erfasst, 
speichert und verwendet. Die zu erfassenden Daten sind in §1g Absatz 1 des 
Straßenverkehrsgesetzes in Verbindung mit Anlage 2 zu dieser Verordnung 
abschließend geregelt." 
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5.2.2 Corner Case Detection 


The data retrieval procedures are needed to make sure that all collected data is 
backed-up and stored in a safe place. The FESTA handbook compares the dif- 
ferent data transfer modes, either manual data ingestion via external hard disks 
and Network Attached Storage (NAS) devices or data transfer via Ethernet 
cable or Long Term Evolution (LTE) connection [FC11]. In the first step, 
the measurements of test CMVs are transferred to an ingest station. For this 
purpose, the geographically distributed data sources are collected according to 
a predefined ingest process [TM20]. The second step is the extraction of corner 
cases from the recorded measurements using data pre-processing methods. The 
data logistics process can either be stored on an on-premise data storage, or a 
cloud-based one. The optimization is executed via pre-processing of the data 
snippets, that are extracted from the computing services (e.g. Amazon Web 
Services (AWS)'), such a technique is illustrated in section 5.3.1. 


Numerous literature on time-series representations is available to facilitate the 
tasks of data search and knowledge discovery [AFS93, ANR74, DTS*08]. 
In addition, the data is converted from the automotive data formats such 
as Measurement Data Format (MDF) [ASA19] and ADTF data format into 
big data file formats such as Comma-Separated Values (CSV) and Hadoop 
Distributed File System (HDFS) [AJK*17]. 


5.2.3 Cloud-based Data Storage 


Statistical statements can be derived from the recorded naturalistic data and 
the real customer behavior can be identified. Meanwhilst, the simulation of 
logged data is a well-known technique for generating virtual open-loop test 
drives based on sensor data collected from field tests. Figure 5.3 shows the 
worldwide FOTs in CMVs over different periods of time. Therefore, the com- 
mon workflow for the development of automated driving algorithms includes 
many iterations of simulations. 


£ https://aws.amazon.com/what-is-cloud-computing/ 
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Figure 5.3: Web-based ODD coverage for worldwide FOTs of an AEB function over various 
periods of time in CMVs [BES*21]. 


Each iteration consists of three steps. First, the relevant input data is selected. 
Second, the playback simulation is performed. Third, the output of the logged 
data simulation is post-processed to evaluate the functional performance of 
the algorithm in terms of KPIs. In such a way, the measurement data of the 
test vehicles is automatically processed and stored on central servers, while 
the meta-information is stored in a database. An adaptable user interface is 
used to visualize, operate and monitor the database and the measurement data 
procedures. Consequently, logged data simulations with different sensor data- 
sets are performed with modified versions of the algorithm to select the most 
suitable version for release in automated CMVs. 
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5.3 Analysis and Triage of Time-Series Data 


The time-series analysis allows the extraction of representative situations ob- 
served during on-road test drives [ESFG18]. The data consists of processed 
object lists of environmental perception sensors that change over time. The 
cluster analysis refers to an unsupervised classification to group the time- 
series data, based on the information retrieved in the data that describes the 
time-series signals and their relationships [W*19]. Consequently, the cluster 
analysis is performed to extract homogeneous groups (clusters) from data-sets 
according to a defined similarity metrics [Pfe20]. 


The time-series within a cluster should be similar to each other and different 
from the time-series in other clusters. In prototype-based clustering, a cluster 
consists of a set of time-series in which each signal is similar to the prototype 
describing the cluster compared to the prototype of another cluster. The ob- 
jective of clustering is to extract clusters from a data-set, where the distance 
between members of a cluster is minimized and the distance between different 
clusters is maximized. The efficiency of the cluster analysis is determined by 
the selection of the time-series to be analyzed, the distance measures to be 
used and the clustering algorithm to be employed. 


5.3.1 Time-Series Data Pre-processing 


The time-series data refers to a specific type of sequential data, where each 
data-set represents a time-series, i.e. a sequence of values that change over time. 
The outliers represent values of time-series data that have some characteristics 
that differ from most other time-series in the data-set. After filtering out the 
outliers from the time-series signals, the rolling standard deviation method 
is used to quantify the degree of variation of each value in the time-series 
and to select the optimized time interval around the triggering event. The low 
standard deviation indicates that the time-series tends to be closed to the mean, 
while high standard deviation implies that the time-series tends to be spread 
over a wider range of values. The determination of the optimized time interval 
depends on the assumption that time intervals with high variations of the sensor 
values are more representative for an efficient cluster analysis of time-series 
signals. Therefore, the interval around the trigger event is selected with a higher 
rolling standard deviation than other intervals. 
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Equation 5.1 represents the normalized rolling standard deviation of each 
point u(a;) of the time-series A= {aj};", to determine the corresponding time 
intervals of the measured sensor variables, where äj+j = 1 diet ai+j indicates 
the rolling mean and w indicates the rolling window. 


Dj (ai+j = Air; )? 


L w 2 
We w2 diel ur 


u(aj) = (5.1) 


The proximity between the two time-series signals is a numerical measure 
of the degree to which the two signals are equal. Proximities are usually not 
negative and are often between 0 for no similarity and 1 for full similarity. The 
time-series proximity can be performed with the Minkowski distance metric, as 
in the equation 5.2 to compare the original time-series with each other, where 
r = 1 for the distance from Manhattan (L; norm), r = 2 for Euclidean distance 
(L2 norm) and r = oo for Supremum distance (Loo norm) [Els18]. 


1 


m r 


D,(A.B) = |), a = bil” (5.2) 


i=1 


Although the Normalized Cross Correlation (NCC) is often used in image 
and signal processing for template matching, similar traffic situations can be 
effectively described by similarities in sensor measurements with time shifts 
and sliding windows. Therefore, the NCC provides as a suitable measure 
for the proximity in the time-series analysis of the measurements from the 
environmental perception sensors. Equation 5.3 refers to the NCC calculation 
between two different time-series A = {aj};", and B = {b;}:", with a time shift 
sj € {8_m,* ++, Sm} and a time interval j € {-m,- - -,m}. 


Sj 1X (ai — a) bij — b 
Dice (AB) = —), > 


i=1 


‚Vi+je[l,m] (5.3) 


Tab 


The mean and standard deviation of the time-series A indicates a = Ł Ži äi 


and o, = 4 | Èr (a- a)’. The mean and standard deviation of the time-series 


B gives b = + X£; bi and o = y £ DF, (bi - 5)? respectively. 
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Based on the equation 5.3, the distance measure De shows a value between 
[-1, 1] for a certain time shift j € {-m,- - -,m} , where —1 indicates a com- 
plete dissimilarity and 1 denotes a perfect match. Equation 5.4 searches for a 
shift with the maximum proximity between the two given time-series over all 
possible shifts and converts the selected shift into a distance measure Dycc. 
The computed distance is then defined in the range [0, 1], where 0 indicates a 
minimum distance for perfect match and 1 indicates a maximum distance for 
complete dissimilarity. 


Dyce (A,B) = = | 1 - max DÑ o(A,B)|,Y j € [-m, m] (5.4) 
J 


NI = 


5.3.2 Principal Component Analysis 


Data-driven test development necessitates continuous extraction of knowledge 
from the recorded sequences. In this context, reducing complexity of data is 
essential in order to perform meaningful analysis [BSMH18]. To this end, PCA 
has been widely used as a simple, non-parametric method of extracting useful 
information from complex data-sets. This principal was defined in chapter 2, 
through subsection 5.3.2, a discussion providing its integration is unraveled. 


The PCA aims to reduce dimensionality variance of data using principle com- 
ponents. This is achieved by orthogonal transformation of the multivariate 
data-set into a new basis which best expresses the data-set. As a case study, 
the PCA is applied to a LDW function in CMVs during the FOTs. The cluster 
analysis of the right side lane departures is based on the lateral distance to 
the right lane dy [m] and takes into account a total of 250 triggered events 
from FOTs, as depicted in figure 5.4. The time-series patterns of dy [m] are 
divided into three clusters, as illustrated in 5.5. The cluster E shows 236 
events for different driving situations with a deviation to the detected right lane 
marking and a return from this deviation. The cluster C illustrates 16 events 
for a number of driving situations with a sudden jump in the distance to lane 
indicating the detection of new lane lines. The cluster C.: shows 3 events for 
driving situations with a deviation and a temporary sudden change of distance 
to the right line and back to normal deviation due to painted islands. 
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Figure 5.4: Cluster analysis of 250 events with right lane departures using the PCA of time-series 
data from dy [m] [ESF19]. 
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Figure 5.5: Time-series data of dy [m] for each observed cluster from 250 events with right lane 
departures using a PCA algorithm [ESF19]. 
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5.3.3 Hierarchical Agglomerative Clustering 


The hierarchical clustering technique is one of the popular clustering techni- 
ques in multivariate time-series analysis and is often represented graphically 
in a cluster-map consisting of a dendrogram and a heat map [E*18]. The 
dendrogram is a tree-like diagram that shows the cluster-subcluster relation- 
ships and the order in which clusters are merged. The heat map represents the 
proximity matrix after the merge operations. The Hierarchical Agglomerative 
Clustering (HAC) produces a hierarchy of nested clusters. The cluster analy- 
sis method follows a bottom-up approach in performing clustering, starting 
at the lowest level where each time-series is a cluster and performing merge 
operations in sequence until one cluster including all time-series is left. 


In addition, the inputs of the HAC algorithm consist of a pairwise distance 
matrix and a linkage criterion to update the matrix during the merge operations. 
Three of the most popular linkage criteria are: Single, average and complete. 
First, the single linkage defines new distances between clusters as the minimum 
distance between any two series in the different clusters when merging two 
clusters. Next, the average linkage defines the distance between two clusters as 
the average pairwise distance between all time-series in both clusters. Finally, 
the complete linkage defines new distances as the maximum distance between 
any two time-series in the different clusters. The choice of the appropriate 
matrix and the corresponding linkage criterion play a major role for the HAC 
algorithm to effectively update the proximity matrix during the merge process. 
Therefore, a complete linkage for the algorithm fnac is chosen to be less 
susceptible to noise and outliers, as shown in algorithm 1. 


Algorithm 1 HAC algorithm with NCC-based distance measure and complete 
linkage. 


E 
p Dyce, complete linkage) 
t= 


1: procedure fyac(G = f0} 
2: while length(G) > 1 do 


3. gD, gO e max Dyce(g®, gO) ¥ gO # g0) 
ge) ge 
G =G \ {8 eu {2 ug”) 


4 
5 return G 
6: end while 

7: end procedure 
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The hierarchical clustering is executed by starting with each element as a 
singleton cluster G and then recursively merging the two nearest clusters g(') 
and g'” until a single cluster remains. As a further case study on cluster ana- 
lysis, the HAC is applied to an AEB function in response to stationary objects. 
The emergency braking operation is initiated to achieve a predetermined target 
safety distance between the Ego-vehicle and the preceding object. Two ob- 
ject list variables measured by the RADAR sensor are selected to characterize 
the event-based driving situations, namely the lateral deviation Sg [m] of the 
relevant stationary object and the predicted road curvature Kego[1/km]. 


The three types of escalation levels where abstractly mentioned in 3, as type 
1, 2 and 3; in which, the first escalation level él, visual and audible alarms 
are emitted as a warning for the driver as long as a relevant object is within a 
predefined distance and closing speed. If the truck driver does not react to the 
haptic and automatic partial braking of the second escalation level €2, and if the 
threat worsens further, then emergency braking occurs in the third escalation 
level &} at a point where the collision is imminent. Figure 5.6 shows the signal 
prototype of selected sensor data variables. The positive curvature keso|1/km] 
values indicate driving in a left turn, while the negative ones indicate right turn 
driving. A positive lateral distance d [m] indicates that the object is on the 
left side of the Ego vehicle, while a negative lateral distance indicates that the 
object is on the right side. 


Figure 5.7 shows a clustermap to visualize the hierarchical clustering of time- 
series events using complete linkage within four clusters of unlabeled trigger- 
events caused by stationary objects with only the escalation levels &! and 
&2. No triggering events were recorded for the escalation level E of the entire 
FOT campaign. The well-separated clusters show a very strong, block-diagonal 
pattern in the reordered proximity matrix. 


The cluster C} displays 179 driving situations with 53% in total of 337 triggered 
events in which the Ego-vehicle triggers a false alarm in a left turn due to an 
irrelevant obstacle on the right lane. The characteristic waveforms of cluster 
C} represent driving in a left turn. This cluster indicates an increasing negative 
lateral distance dy [m] to the object in front, which indicates that the truck is 
moving to the left. The driving in a left-hand curve is also evident from the 
increasing value of road curvature positively. 
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Figure 5.6: Characteristic waveforms of the trajectories based on the road curvature of the Ego- 
vehicle Kego[1/km] and the lateral distance of the stationary object in front of the 
Ego-vehicle de [m] for each cluster [ESFG20]. 


The cluster C? indicates 58 driving situations with 17% in which the Ego- 
vehicle erroneously triggers a warning with an unrelated obstacle located on 
a left-hand traffic island. This cluster shows events where the truck has driven 
to the right and subsequently to the left, basically driving around the object 
from the right-hand side. As a result, characteristic signal courses of the cluster 
C show driving around an object from the right-hand side. Furthermore, the 
cluster CÌ collects 32 driving situations with 10% in which the Ego-vehicle 
triggers a false-triggered event with an unrelated obstacle located on a right- 
hand traffic island. The cluster C? demonstrates that trucks drove around an 
object from the left-hand side. Eventually, the cluster C* represents 68 driving 
situations with 20% in which the Ego-vehicle triggers an inappropriate warning 
in a right turn due to an irrelevant obstacle on the left lane. The cluster C4 
indicates events where trucks are driving in a right curve. 
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Figure 5.7: Clustermap of hierarchical clustering with complete linkage from 337 driving situa- 
tions using the HAC of time-series data from Kego[1/km] and qi! [m] [ESS*19c]. 
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Figure 5.8 illustrates the four clusters, which cause false warnings of the AEB 
function by detecting bounding objects of the road as relevant objects, and 
which can be identified by trigger conditions. In the highway engineering, the 
transition curve is an essential design element in the horizontal projection of 
the road design besides the straight line and the circular arc. The transition 
curve is a curve in which curvature Kroaa[1/km] varies uniformly with respect 
to its length. It allows a gradual change from one radius to another or from 
a straight line to a circular curve, since a straight line is merely a curve of 
infinite radius. Therefore, a circle has a constant curvature and a straight line 
has a curvature of 0. The most common transition curve is the clothoid, which 
ensures a smooth transition between the horizontal alignment elements with a 
constant curvature. As a transition curve type, the combined curve follows the 
sequence (clothoid - circular arc — clothoid). As a further type of the transition 
curve, the S-shaped clothoid, also called inflection line, consists of two clothoid 
branches with curvature in opposite directions [Kühl3]. 
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Figure 5.8: Characterization of 337 driving situations on the basis of recorded sensor signals for 
environment perception and their system reactions during the FOTs [BES*21]. In real- 
world footage (Source: documentation camera). 
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As expound in chapters 2,3 and 4, the vision of accident-free driving acceler- 
ates the development of intelligent, connected and complex driving functions 
in CMVs. The conventional test methods pose two major challenges for the 
testing of ADFs. The numerous combinations of relevant driving scenarios 
present a challenge to meet a complete set of functional requirements. On the 
other hand, ADFs employ machine learning algorithms with opaque functional 
requirements. As discussed in chapters 3 and 4, the release of automated CMVs 
requires not only comprehensive testing against realistic driving scenarios, but 
also the KPIs to evaluate the ADFs with respect to dealing with uncertain- 
ties [R6s20]. Consequently, the further development and market introduction 
of ADFs have led to a need for new methods of software development and 
evaluation [HS10]. 


The road traffic expresses an open parameter space in which an infinite 
number of different traffic situations can occur. But it is also not possible 
to guarantee absolute safety for automated CMVs. Due to the complexity 
of the development of ADFs, SOTIF recommends continuous improvement 
iteratively, so that the residual risk can be accepted [SH19b]. Although a 
human driver doesn’t drive perfectly at times, especially during the first few 
driving lessons, the development of his predictive mental model enables him 
to expand his accumulated driving skills. Therefore, the software product lines 
have established knowledge-based and data-driven test methods to ensure the 
functionality of their products in terms of robustness, reliability and safety. 
Moreover, a functional decomposition is necessary to present appropriate ar- 
guments by measuring and addressing the residual risks caused by deficiencies 
in the environmental perception sensors [Hüll8]. Accordingly, the measurable 
safety framework provides a data-driven test method that complements the 
knowledge-based test method and adaptively enriches test cases with an ODD 
coverage. 
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6.1 


There are numerous steps in the data mining process to ensure continuous mon- 
itoring and learning from field observations. For this purpose, the measurable 
safety framework aims at bridging the gap between the knowledge-based and 
the data-driven test methods by continuously expanding knowledge in an ODD 
coverage. In addition, the framework provides a meaningful termination cri- 
teria for testing the ADFs based on the performance evaluation of individual 
components as well as that of the entire system. Figure 6.1 schematically 


Measurable Safety Framework 


illustrates a test method using the measurable safety framework. 


Figure 6.1: Overview of the measurable safety framework with its main elements by means of test 
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termination criteria and ODD coverage [ESS* 19c]. 


6.1 Measurable Safety Framework 


All ADFs are required to be compliant with the ISO 26262:2018 for functional 
safety and shall not adversely affect each other. Consequently, the ADFs need 
to be adequately safeguarded to ensure that the functional safety requirements 
of ISO 26262:2018 are met, in particular with regard to unintended interven- 


tions. 


The following steps are obligatory steps performed in the test method 


to evaluate the optimization quality of ADFs within the ODD of the vehicle. 
They do not need to always be in same sequence as mentioned below, their 
order can vary as per requirement. 


(1) 


(I) 


(IIT) 


(IV) 


(V 


Nat 


(VD 


(VID 


Verification of the absence of logical errors for the ADF by leveraging 
the HiL test bench according to the knowledge-based requirements, as 
presented in section 6.2. 


Identification of statistical errors within the ADF by scenario-based 
testing using cluster analysis of time-series data in a cloud database, as 
shown in section 5.3. 


Determination of criticality metrics for the ADF through correlation 
analysis between the criticality threshold of synthetic driving scenarios 
and events from the various clustered naturalistic driving situations, as 
displayed in section 6.4. 


Execution of test cases on the HiL test bench for the extracted logical 
scenarios with the ontology-based scenario management after converting 
the open-loop detection data to closed-loop control data within the ODD 
coverage, as presented in section 6.3 


Exploration of the parameter space for the observed logical scenarios 
using sensitivity analysis to generate an approximation model of the 
parameters under consideration, as demonstrated in section 6.5. 


Prediction of the risk acceptance threshold as a confidence level using 
reliability analysis to estimate the probability of exceeding the safety 
margin using various sampling methods, as exposed in section 7.2. 


Definition of the test termination criteria on the basis of the generated 
tolerable risk curve as reference safety threshold for the further develop- 
ment of the ADFs and the decision whether more kilometers are needed 
or more simulations have to be carried out, as shown in section 7.1. 
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Another approach to account for additional field data is specified as an optional 
method step: 


(VII) Comparison of the additional field data with observed logical scenarios, 
possible extension of the number of clusters, repetition and application 
of the steps (ID to (VID). 


6.2 Demonstration Case Study 


The AEB function comprises a detection unit to determine the distance 
and speed of the Ego-vehicle relative to the leading object or obstacle. The 
emergency braking procedure is initiated in steps to achieve a predetermined 
target safety distance between the Ego-vehicle and the object or obstacle in 
front. The target safety distance corresponds to the latest time at which full 
braking must be initiated to avoid a collision. As per seen in chapters 3 and 
4, the process of bringing a CMV to standstill requires a complex interaction 
between the brake actuator, the CMV tires, the CMV dimensions, the load 
conditions and the road surface. 


False interventions can be avoided or at least significantly reduced by the de- 
tection of edge structures and boundary objects of the road (e.g. guideposts, 
crash barriers, traffic signs, etc.). Such constructions and objects depend on 
the road type and are taken into account when triggering the escalation levels 
&!, &2 and &? by adapting the triggering conditions to the road classification. 
The ISO 22839:2013 specifies a test method to define the minimum func- 
tionality requirements of the AEB system with respect to FP and FN events. 
The scenario-based test concept emphasizes the testing of particularly critical 
traffic scenarios. Most typical traffic situations are not regarded as particularly 
dangerous and therefore contribute relatively less to the PoS. The identification 
of critical scenarios therefore requires indicators that quantify the criticality of 
traffic situations [Sch17a]. The deterministic TTR indicators are used, which 
describe the remaining time until a critical event occurs, such as Time Head Way 
(THW) and TTC. The THW describes the required time for the Ego-vehicle to 
reach the current position of the relevant object, as illustrated in equation 6.1. 


rel 


d 
THW = a LV ue > 0 (6.1) 
x 
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The TTC describes the time at which a collision occurs, if Ego-vehicle and 
detected object don’t change their speed and direction. Nevertheless, TTC is 
used as a generic KPI for the functional evaluation even with constant relative 
speeds and accelerations. Equation 6.2 presents the mathematical description 
of the TTC at a constant relative velocity between the Ego-vehicle and the 
stationary object for each escalation level (&!, &2 and 8?) with emergency 
braking of the Ego-vehicle until standstill. 


rel 


d 
TTC =~ ,V af! =0, ux >0, df >0 (6.2) 
Ux 


In case of a constant relative acceleration between the Ego-vehicle and the pre- 
ceding object for each escalation level (€!, &2 and &?) with emergency braking 
of the Ego-vehicle until standstill, the equation 6.3 gives the mathematical 
description of the TTC, as follows: 


—yrel + viel 2 +2% are! $ rel 
TTC = Er ‚Vael<o, v> 0, dso 
(6.3) 
The developed test method executes step (D by elimination of logical errors 
through verification of the AEB function using the HiL platform — briefed 
in chapter 3, and detailed in chapter 4 — according to the knowledge-based 
requirements, as depicted in figure 6.2. The logical errors represent a subset of 
the discrepancies between the specified and implemented behavior of the AEB 
algorithm. The simulation results rely on synthetic data from a camera ECU 
and aRADAR sensor model, which are applied to the HiL test bench. 


The TTC parameters are determined on the basis of the temporal progression 
of the Ego-vehicle’s braking when approaching a stationary object at different 
longitudinal velocities v!°'[km/h] at the various escalation levels (&!, &2 and 
&2). If the braking cascade of AEB controller is triggered when approaching a 
stationary object, the minimum TTC can be evaluated accordingly. Therefore, 
the pass/fail criteria use criticality parameters such as THW and TTC for 
the criticality assessment of the test results [JBKW18]. Other deterministic 
indicators use vehicle dynamics parameters and physical capabilities of the 
CMV to assess criticality, such as the required braking acceleration. 


The deficit of these indicators is their reliance on the assumption of proper 
trajectory prediction. Consequently, small changes in motion prediction can 
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Figure 6.2: Identification of the TTC parameters using the HiL platform by scenarios approaching 
a stationary object with a straight road at different longitudinal velocities of the Ego- 
vehicle [ESS* 19c]. 


lead to significantly different results [W* 16]. In step (ID, the measurement data 
from country-specific field tests are recorded and stored centrally in a cloud 
database. Then, the cluster analysis takes place to identify the FP and FN errors 
(elucidated in chapter 2) from the field observation. These error types occur due 
to environmental influences during operation and are statistical in nature for 
the discrepancy between the intended and specified behavior. The criticality 
thresholds are derived in step (III) from the simulation results using an ap- 
propriate regression function. Subsequently, a correlation analysis is applied 
between the proposed criticality threshold from synthetic driving scenarios and 
events from the individual grouped naturalistic driving situations. In step (IV), 
the HiL platform employs systematic test case generation to extend the test 
coverage within the applied ODD using ontologies and equivalence classes. 
The conversion rules derive the control data of concrete scenarios from the 
characteristic waveforms of the acquired sensor signals from the FOT. 
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6.3 Test Case Generation 


Figure 6.3 illustrates exemplary time-series of the detected sensor signals of 
the cluster C}. The cluster C} presents driving situations in a combined curve 
to the left, where the predicted road curvature Kego[1/km] is increased. At 
the same time, the lateral deviation to the stationary object in front d\"'[m] is 


decreased. 
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Figure 6.3: Event-based data acquisition of the curvature of the Ego-vehicle Kego[1/km] (left) and 
lateral offset of the stationary object in front di! [m] (right) for the cluster C [ESD20]. 


On the basis of the stochastic variation within the parameter space ofthe logical 
scenarios, the process of generating the concrete parameter sets for the ontolo- 
gies is ensured. Thereby, the identification of a concrete set of parameters 
within the ODD coverage follows the search for critical parameter sets within 
the parameter space. Accordingly, each concrete parameter set corresponds to 
a concrete scenario and vice versa. 
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Figure 6.4 depicts a logical scenario from the cluster C}. The characteristic 
waveforms of cluster C! represent driving in a left turn, as seen in figures 5.6 
and 5.8. The ontology uses a [consists_of] statement to model the elements of a 
road network layout with two lane classes and one class for a hard shoulder. The 
statements [has_right_neighbour] and [has_left_neighbour] are used to arrange 
road elements to each other. The position instances are generated on the basis 
of a relation [offers_position] for the ontology road elements. The statements 
[left_of], [right_of], [front_of] and [rear_of] are utilized to arrange the position 
instances with logic reasoning. The statements [driving_on] and [located_on] 
are employed to control the dynamic objects with different position instances. 
The object [obj1] is defined as a stationary object on the hard shoulder. The 
ontology specifies axioms that define constraints on attributes and relationships 
for specific concepts. 


The semantic transformation rules utilize the open-loop sensor signals acquired 
by the ADDR to generate the respective closed-loop control data within the 
scenario synthesis process. These rules are applied, for example, to express 
the obtained curvature data Kego[1/km] as Ego-vehicle trajectory data within 
Cartesian co-ordinates for the concrete scenarios. The direction angle w;[°] is 
measured between the tangent in the current position (i) of the clothoid and 
the initial direction with (K;oaq~ 0). The formula [wi = 0.5 * Li * xj] calculates 
the direction angle w;[°] based on the length of the clothoid arc to the current 
point L;[km] and the current curvature «j[1/km] [Kol10]. Equations 6.4 and 
6.5 calculate the x;[m] and y;[m] in Cartesian co-ordinates as follows: 


_ Ri i ww a 
Aay a VO Bart Oat en 
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The OWL is implemented in the ontology for each logical scenario, that allows 
the Semantic Web Rule Language (SWRL)! to combine logic operators into 
rules. Therefore, invalid or forbidden combinations can be eliminated from the 
scenario catalog. 


i https: //www.w3.org/Submission/SWRL/ 
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Figure 6.4: Logical scenario synthesis of cluster C! when driving on a left curve during coming 
close to a stationary object with 179 events [ESD20]. 


137 


6 Operational Design Domain Coverage 


The SWRL rules are included in the ontology file? to extend the axioms of the 
logical scenario by inferring knowledge [UNGO21]. The context-driven test 
concept requires the overall probability Pr that an AEB function is faulty and 
executes asetofscenarios Fr in which the AEB function performed incorrectly. 
Since the logical scenario Fc for combined curves presents clusters C} and 
C3 as a set of scenarios in which the Ego-vehicle drives in a combined curve 
with a stationary object on the side, its conditional probability is determined by 
P(Fr|Fc) as probability that the system operates erroneously when driving in 
a curve with a stationary object on the side. P(Fc) represents the probability, 
whereby driving in curves with the object standing sideways occurs in all traffic 
scenarios. If the clusters C? and C} have to be considered, the ontology has 
to be expanded with the functional scenario Fs for S-shaped clothoid curves. 
The combined curve consists of a sequence of a clothoid, then a circular, and 
then a clothoid again. Meanwhile, the S-shaped clothoid consists two clothoid 
branches with curvature in the opposite direction and abutting at their zero 
point [Kiih13]. 


Since the logical scenario Fs describes the set of scenarios in which the vehicle 
goes around a stationary object, its conditional probability is determined by 
P(Fr|Fs) as probability that the system operates erroneously when driving 
around a stationary object. P(Fs) represents the probability, whereby driving 
around a stationary object occurs in all traffic scenarios. Equation 6.6 presents 
the overall probability Pr to fulfill the functional scenario catalogs extracted 
from cluster analysis of triggered events for an AEB function. 


Pr = P(Fr) = P(Fr|Fc) * P(Fc) + P(Fr|Fs) * P(Fs) (6.6) 


In general, a functional scenario can be defined as a combination of logical 
scenarios U = [Ul, Us, -- , Un] of n-dimension. The overall probability of 
functional scenarios can be generally formulated as in equation 6.7. 


Pr = P(W) = Y POT + PC) (6.7) 
i=1 


2 https://protege.stanford.edu/ 
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6.4 Correlation Analysis 


The curve fitting process uses an appropriate regression function to generate an 
ideal criticality threshold from the simulation calculations applied to the HiL 
test bench. While the variable slope sigmoidal equation is often used for a re- 
gression analysis of pharmacological dose-response curves, the four-parameter 
logistic regression is generally also employed for curve fitting analysis. The 
dose-response curve takes the form of a sigmoid curve that is shaped like the 
letter S. 


Equation 6.8 uses the four-parameter logistic nonlinear regression to fit the 
curve of the TR, [s] measurements, where P1 denotes the baseline response, 
P2 the maximum response, P3 the turning point of ue" [km/h] giving a halfway 
response between the baseline and maximum, and P4 the curve slope. The 
TTEA! [s] refers to the ideal criticality threshold of the TTC for the escalation 
level &! based on synthetic data using driving scenarios approaching of a 
stationary object with a straight road at different longitudinal velocities of the 
Ego-vehicle. 


P2 -P1 P2 -P1 
TTC? = Pl + ———__,, =Pl+ ——_,,, JV yel >0 (6.8) 
j 1+ | 10P3 |" 1 + 10(P3-ux')*P4 


The sigmoidal dose-response equation extracts the logistic curve parame- 
ters based on the THe As) measurements from the HiL test bench, where 
(P1= 0.52, P2= 4.24, P3= 37.6 and P4= 0.02). Subsequently, the correlation 
analysis measures the statistical relationship between the FOT data and the 
ideal criticality threshold to a standardized covariance ranging between —1 
and 1. The direction and strength of the linear dependence between the two 
time-series are quantified by that coefficient. The estimated Pearson Product- 
Moment Correlation Coefficient (PPMCC) becomes more inaccurate, as its 
value is closer to zero. If both variables have a strong positive correlation, the 
PPMCC is close to one, for a strong negative correlation is close to minus one. 
The linear dependence between TICA [s] and ae, [s] can be expressed as 
ratio between the covariance and the product of standard deviations. 
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The TICH [s] designates the TTC of the escalation level &! for the cluster 


C! events with the combined curve to the left. Equation 6.9 calculates the 
PPMCC = 0.8055, where cov(TTCE! TTCS) refers to the covariance, Oc! 
ref 


ref’ fot 
is the standard deviation of TC: [s] and o-rcs: is the standard deviation of 
fot 


TIC. [s]: 


fot 


cov(TTCE!,TTCE!) 
PPMCC = 1) = en “ 
P (TTCEL,TTCE!) CTrcE! * Tyce o? 


Equations 6.10 and 6.11 calculate the covariance and the standard deviations 


for TTCS! [s] and TICS: [s], where Hrrcë! and Myce! are the estimates of the 


mean values, respectively. 


ref? fot ref 


n 
cov(TTCÊ;, TICE) =) (TICE! - urrce:)(TTCÊI — Merce) (6.10) 
i=l 


2 2 2 2 
Ortes = \ I, (TTCH = rres) rnc = \ D, (TICE <Anrett) 
i=1 i=l 
(6.11) 


The coefficient of determination, denoted R?, is the square of PPMCC with 
(R?= 0.6488) and evaluates the strength of the linear relationship between 
the two time-series TTC®![s] and TTC®! [s]. While the PPMCC and R°? val- 


ref fot 


ues show a strong positive correlation between me [s] and re. [s], the 


equation 6.12 calculates the de [s], which is assumed as a quantifiable error 
indicator of the environment perception. 
öl — él él 
dire = TTCH — TTC (6.12) 
The TTE parameters of triggered events from FOT can be correlated with 
the pass/fail criteria obtained from the HiL to identify the criticality of each 
triggered event. Figure 6.5 shows the parameter identification of dÊ} [s] based 


TIC 


on the correlation and regression analysis between TEO criticality threshold 


and the TiC! for 179 events of cluster C at the escalation level &!. 
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Figure 6.5: Correlation and regression analysis between digital and physical tests with the example 
of the TTC criticality assessment [ESD20]. 


To avoid accidents, CMVs need greater distance and time to come to a complete 
stop compared to passenger cars. The braking distance of a truck is even longer 
when it is hauling a heavy load and/or there are adverse road conditions such 
as snow, ice or rain, as demonstrated in chapter 4. At the same time, the 
AEB function requires to operate in the robust detection ranges, so the Te 
curve saturates at higher velocities of vê! above 50[km/h] to keep the reaction 
distance increasing linearly with the v’°' and thus optimize the sensitivity and 
specificity of the AEB. The mr. is calculated by the HiL test platform 
based on synthetic sensor models, whereby the simulation environment is 
communicating with the embedded ECU for AEB function under real-time 


conditions. The det os] represents the subtraction between TTE: [s] and 
TIO [s] values and is expressed as a numerical value denoting a quantifiable 


error indicator. 
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6.5 Sensitivity Analysis 


Saltelli et al. [S*08] defines the sensitivity analysis as a study of how the 
uncertainty in the output of a model or system can be apportioned and allo- 
cated to different sources of variation in its inputs. After defining the input 
variables and model responses, the sensitivity analysis then scans the param- 
eter space with Design of Experiments (DoE) or sampling methods. Ebert et 
al. [EGK*20] applies the sensitivity analysis to reduce the dimension of the 
DoE to the most important factors as a part of the engine calibration process 
within the Real Driving Emissions (RDE) test cycle. Accordingly, the sensi- 
tivity analysis can be used to analyze the contribution of each input variable to 
the scatter of each model response. Each scattering input specifies a distribu- 
tion type including mean value and variance, whereby the input variables are 
defined as random variables. The dependencies between the input variables 
can be formulated in terms of linear correlations. The meta-models are also 
used to represent the model responses by surrogate functions in terms of 
the model inputs [BSH17]. Meta-model approaches such as Kriging approx- 
imation, Metamodel of Optimal Prognosis (MOP) approximation, Support 
Vector Regression (SVR) and Artifical Neural Network (ANN) provide an 
automatic reduction of the variable space. The meta-model applied in this 
work is based on the anisotropic kriging approach using the Metamodel of 
Optimal Prognosis (MOP) solver [MW08]. 


Most et al. defines the coefficient of prognosis as a sensitivity measure to 
determine the optimal input variable subspace together with the optimal meta- 
model approach. The sensitivity analysis is applied in the example of the 
cluster C} at the escalation level &!. The sensitivity measures are determined 
in step (V) using the sensitivity analysis, whereby the input variables of the 
approximation model are (vË! [km/h],d°'[m], d°! [m], K&L [1/km] and dacs) 
and the approximation model output is TIO [s]. The scalar random variables 
can be used to model the scattering inputs. The MOP solver reduced the input 
variables from [vf ‚dd dË! KEL, dio] into [vf dEi]. While it is important 
a was eliminated by the MOP solver, although 
curvature could play a role in determining 7 for such FP events — as 
shown in the figure 6.4 — , it did not, arguing that correlation does not imply 
causation. 


to note that the variable « 
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Definition 6.1 (minimum TTC):It describes the lowest time-to-collision in 
which an environmental perception sensor reports a tracked object list with 
objects classified as relevant for the ADF. The minimum TTC relates to the 
error-object classification in an inverse-proportional sense; the lower the value 
of THe the more prominent the error in classifying the object for such FP 


events in the cluster C? [BES*21]. 


6.6 Exploration of Parameter Space 


In general, the mean value and the standard deviation are used to describe the 
variation of arandom variable. However, the shape of the distribution function 
can have a significant influence on the results of a stochastic analysis. Accord- 
ingly, the parameter space exploration requires the definition of distribution 
types including mean value and standard deviation for all random input vari- 
ables. The truncated normal distribution is chosen as a suitable distribution type 
for the measurements of Ego-vehicle curvature kel 1/km], lateral deviation 
to the stationary object a [m] and error indicator of environment perception 
aĉi [s] at the escalation level &! of the cluster C}. The truncated normal 
distribution is a normal Gaussian distribution that is restricted to lie within a 
finite range, i.e. [Kann < < Kego |. The truncated normal distribution can 
be expressed in terms of the normal Gaussian distribution as follows: 


él 
Kego H 
öl. „min „max 


= min él max 
JPDF (Kego: Kego » Kego » H» o) = Tak K <K SK 


min _ » “ego ego — “ego 
BES 
(6.13) 
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Kel u min _ y 
6&1. „min „max ( z ) u D( = ) 
Sevr (Kogo; Kego> Kego » H» o)= 


ot) — (SS) 
(6.14) 


where u and o denote the mean and standard deviation of the parent normal 
distribution and «2)"= —4.5[1/km] and Kego = 3.9[1/km] as the lower and upper 


ego 
truncation points respectively. 


143 


6 Operational Design Domain Coverage 


The Q and ® designate the Probability Distribution Function (PDF) and the 
Cumulative Distribution Function (CDF) for the normal Gaussian distribution 
respectively. Figure 6.6 illustrates the PDF and CDF estimations using the 
truncated normal distribution. Equations 6.15 and 6.16 estimate the mean and 
standard deviation of the truncated normal distribution. 


nf PDF ---0---- CDF 


Figure 6.6: Estimation of the PDF and CDF of the Ego-vehicle curvature x, [1/km] (top) and the 
lateral deviation to the stationary object aĝ! [m] (bottom) at the escalation level &l of 
the cluster C. 
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The same calculations are applied for the lateral deviation to the stationary 
object dê ! [m], where dein —2.6[m] and dy = 0.8[m] as the lower and upper 
truncation points respectively. The false warnings from the cluster C} are 
presented as an example for the sensitivity analysis. Accordingly, the sensitivity 
measures are analyzed to define the important input variables in relation to the 
response variable TICE! [s]. 


mın 


The triangular distribution is chosen as a suitable distribution type for the 
measurements of the Ego-vehicle longitudinal velocity v?'[km/h] and the 
minimum time to collision TTC is] at the escalation level é! of the cluster 
C!. The following equations calculate the PDF, nn mean and standard 
deviation of the Ego-vehicle longitudinal velocity vE! [km/h] based on the tri- 
angular distribution respectively, where v®™ = 7.9 [km /h], v? = 83.4[km/h] 
and uP _ = 59.6[km/h] as the lower limit, upper limit and moi respectively. 
Equations 6.19 and 6.20 estimate the mean and standard deviation of the 
triangular distribution of the ye 'Tkm/h]. 


2[ vl- arn] an él peak 
[ max min II peak ming? Yu < Vx < Ux 
fi (vel; ymin ymax peak) _ Uk Ux Uk — Ux 
PDF\Yx »%x Ux »% = i 7 
Loy or] y peak 8l < „‚max 
[ums ymin] [ymax _ „Peak 2 Ux < Ux = Ux 
x x x 
(6.17) 
é1 min]? 
[ver] min él peak 
. È [umumin] [urk umin] $ Vu = Ux = Ux 
feor(ve!; ymin „max u ) = x x x x 
xx »°x > 
max E172 
= k 
a max er a peak „Vor“ < yel < ae 
lvo] (6 18) 
1 ; k 
= min max peak 
Hv = z [Vx tU, + Ux ] (6.19) 
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The same calculations are applied for the minimum time to collision TTCä, Is] 
on the basis of the triangular distribution function. Figure 6.7 shows the PDF 
and CDF estimations of the v! [km/h] using the triangular distribution and the 


dels! using the truncated normal distribution. The calculations are applied 


for the error indicator of environment perception dêl [s], where the minimum 
value of de = -1.5[s] and the maximum value of dr, = 0.14[s] are the 


lower and upper truncation points respectively. 
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Figure 6.7: Estimation of the PDF and CDF of the error indicator of environment perception 
dêl Es] (top) and the Ego-vehicle longitudinal velocity ye! [m] (bottom) at the esca- 


lation level E} of the cluster CL. 
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7 Reliability Analysis Using 
Sampling Methods 


The ADFs in a vehicle have to comply with the ISO 26262:2018 standard 
for functional safety and are not allowed to influence each other negatively. 
The Regulation No. 79 applies to the steering equipment of CMVs for lane 
change operations. Meanwhilst, the ACSF of Category E requires minimum 
monitoring ranges for the front, rear and side detection of automated vehi- 
cles [EST*17]. If a motorcycle approaches the Ego-vehicle from behind, the 
rear detection has to take place early enough to prevent the motorcycle from 
braking abruptly when the Ego-vehicle changes lanes. If the distance is below 
a critical range, the lane change request must be suppressed. At the same time, 
the CMVs include a variety of series and models with tractors, semi-tractors 
or trailer combinations. Consequently, the tractor-mounted environmental per- 
ception sensors cannot provide a full 360° surround-view when a trailer is 
coupled to the tractor. In Germany, according to Section 4 of the German road 
traffic regulations, the Ego-vehicle in front must not brake suddenly without 
a compelling reason. Therefore, compliance with ISO 26262:2018 is achieved 
if an ADF represents an acceptable residual risk for the subsequent traffic. 
A probabilistic safety analysis is needed to quantify the uncertainties and to 
enable a prospective risk assessment for the testing and further development 
of ADFs [BRW*17]. 


7.1 Probabilistic Safety Assessment 


Following the application of cluster and sensitivity analysis to the real traffic 
data-set — interpreted in chapters 5 and 6 — , the data is then used to deter- 
mine the probability distributions of the input variables and the probability of 
occurrence of the response variables [Mul 18]. 
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In accordance with ISO 26262:2018, the reference values of the tolerable risk 
curve are derived in step (VT) as a confidence level for the optimization quality 
of ADFs. The measurable safety methodology aims at evaluating the SOTIF- 
or OEDR-related capabilities of ADFs. Accordingly, the reliability analysis 
aims at identifying the probability of safety violation for each logical scenario 
[AKK* 19]. The parameter space is searched with a suitable sampling method to 
determine the probability that a predefined safety margin is exceeded [Roo02]. 
The scattering input variables are considered as random variables within the 
framework of probabilistic safety analysis. Therefore, the probability of failure 
P; is defined as the probability of the event that a random vector Z falls into the 
failure domain. The failure mode involves defining a failure criterion known 
as the Limit State Function (LSF) or safety margin. A direct formulation of the 
LSF in dependence of the input random variables is often not possible, but is 
implicitly given by the simulation results of the investigated system [Bay99]. 
The LSF forms the basis for the reliability calculations and denoted by g(Z). 
The function expresses Resistance-Load as a function of Z, where Z is a vector 
of all uncertainty variables describing the loads and resistances. The failure 
criterion is consequently defined as g(Z) < y, where y denotes a predefined 
safety margin. For a given LSF with g(Z), the probability of failure P; is defined 
as follows: 


P; = Plg(Z) < 7] = J aioa J POR (7.1) 


where {z € R"|g(z) < y} represents the failure domain, (z) designates the re- 
alization of Z in the variable space and fz(z) denotes the joint density function 
for Z. Accordingly, the probability of failure is the integral of the joint PDF over 
the failure domain. While the probability of failure is the complementary of 
reliability, the failure criteria do not have to indicate a system breakdown. Such 
failure criteria can be defined by the violation of quality or safety requirements. 
The probability integral can be expressed as the expectation of the indicator 
function /(g(z)), where /(g(z))= 1 if g(z) < y and /(g(z))= 0 otherwise. The 
probability integral can be interpreted as the expected value E of the indicator 
I(g(z)) as follows: 


Pr = E[i(g(2))] = / = l 1(g(z))fz(2)dz (7.2) 
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While the space of allrandom variables can be separated into asafe domain and 
a failure domain, the indicator’s expected value E[/(g(z))] calculates the prob- 
ability of severity violations within the prospective risk assessment. The relia- 
bility analysis identifies the failure region in the parameter space to predict the 
probability of failure for each logical scenario. While the analytical calculation 
of expected values E[/(g(z))] with the dimensional increase of input random 
variables is no longer suitable for practical use, the Monte Carlo integration 
approximates the indicator’s expectation E[/(g(z))] by simulation-based ap- 
proximation. The simulation of rare events approximates the integral solution 
using Monte Carlo Sampling (MCS) and Adaptive Importance Sampling (AIS) 
methods [WPZ19]. In compliance with the HARA of the ISO 26262:2018, the 
failure criteria present the different levels of severity [LPP11], whereby the 
TRE?! Ís] defines the different severity levels in an exemplary way. 


The TICA. describes the lowest TTC in which an environmental perception 
sensor reports a tracked object list with objects classified as relevant for the 
AEB function of the cluster C} at the escalation level &!. The sensitivity 


analysis defines the variables ve 'Tkm/h] and dé [s] as the most important 


factors influencing the variable Tie [s] as a response within the probabilistic 
safety assessment. The selection is determined by the variables with the greatest 
impact on the model output TTCË! [s]. Accordingly, the reliability analysis is 
applied to the inputs and response of the cluster C. at the escalation level &!. As 
discussed in section 5.3.3, the &! events represent phantom warnings, where the 
situation was safe and a warning was erroneously signaled. The prospective 
risk assessment R (c?) is defined as the combination of the probability of 
occurrence P(C!) of harm and the severity of that harm. Table 7.1 shows the 
severity levels, which are divided into four regions as follows: 


Severity level condition Hypothetical severity level 
so =i] TIC: < 0.5s] critical severity 
S? = [0.55 < Ir, < Is] high severity 
Te &l 3 = 
S; = [ls < TTC in < 2s] medium severity 
Ss? = [ TTCÊ! > 2s] low severity 
min 


Table 7.1: Hypothetical severity levels for object classification error based on the criticality indi- 
cator TICs: [s]. 
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Equation 7.3 represents the prospective risk for the cluster C with the com- 
bined curve to the left. In step (VID, the tolerable risk curve is generated as 
a reference safety threshold with respect to the logical scenarios. Thereby, the 
logical scenario with combined curve to the left is shown as an example in 
order to prospectively estimate the risk regarding false alarms on subsequent 
traffic. 


R (cl) = [PECS | CH) * PCCD] + [P(S21C) PC) + 
[Pe(S; | CE) * P(CH)] + [PS Il») 7.3) 


Figure 7.1 represents the PDF and CDF estimations of the minimum time to 
collision TTC [s] using the triangular distribution function with the corre- 
sponding severity levels. The TTE [s] is used as a prototypical safety measure 
for the reliability analysis. The original measurements of TTO range between 
0.03[s] and 2.27[s]. The triangular distribution is selected to approximate the 
m distribution, where the lower bound of TTC = —0.3[s] and the upper 
bound and mode of me = 1.91[s]. 
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Figure 7.1: Estimation of the PDF and CDF of the response variable TTCË! [s] of the cluster CG 
at the escalation level &!. 
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7.2 Monte Carlo Simulation 


The MCS method estimates P; by generating independent samples from the 
PDF of fz(z) and taking the average of the sample indicator values as an 
unbiased estimator of the expected probability of failure [Zio13]. While the 
joint density function fz(z) depends on random input variables, the estimator 
of its probability failure Pp is also a random variable [Buc09]. Equation 7.4 
represents the probability failure estimator P; as follows: 


Nme 


Pr = IEO) = SIE) 14) 
me 42] 


The estimator variance Var(P;) can be computed approximately from the gen- 
erated samples. In accordance with the central limit theorem, the standard 
deviation converges towards 0 for a large number of samples Nmc— ov. If 
the estimator’s variance converges towards 0, the confidence interval of the 
estimator becomes narrower and thus the estimator gets more confident. The 
number of samples is used to normalize the standard deviation of the unbiased 
estimator P; of the target failure probability P; [P*14]. The normalized stan- 
dard deviation is used as measure for the completion of the required number 
of generated samples and is known as standard error e5,. 


= (7.5) 


The MCS method evaluates the probability of failure by determining whether 
the LSFs are exceeded. The trial is repeated many times to ensure the con- 
vergence of the statistical results. In each trail, a sample value is generated 
and evaluated as a Hit or Miss (relevant or irrelevant) according to the LSF 
definition. The MCS requires a large number of samples to accurately predict 
the probability of failure, especially if the expected value is small. Figure 7.2 
illustrates the result of MCS method used to obtain the failure probability of 
the severity level S?. As shown in figure 7.2, the failure probability Pp(S?|C_) 
and the standard error ep, (S IC!) are plotted against the number of samples 
Ne: 


151 


7 Reliability Analysis Using Sampling Methods 


The MCS sampling is a Hit-or-Miss sampling method. Here, the blue dots indi- 
cate the target Hit samples at which the corresponding LSF condition for each 
severity level is met. Meanwhilst, the yellow spots show the Miss condition, that 
means they don’t fulfill the conditions required for the severity level. The Hit- 
or-Miss MCS method generates Nme samples with 1000 (assumed budget of 
samples), where the number of Hits is 99 and the number of Misses is 901. The 
failure probability P;(.S3|C!) is calculated to be — = 0.1. For this severity 
level, the Hit samples are very little compared to Miss samples and are located 
for the lower values of yS l [km/h] towards the left end of the plot. The target 
is missed for the values of oe higher than 50[km/h] (In which u indicates 
the velocity at escalation level 1). The footage representing such a scenario — 
cluster 1 — is depicted in chapter 5. The stopping criteria of simulation depend 
on the comparison between the standard error of the exceedance probability! 
ep, (S IC!) = 0.0094 and the target standard error dp, = 0.05. 


a Hit o Miss me P;(S3|C}) =-=- ep, (S3|C2) 
1 - i - 


0.8 + 
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Figure 7.2: Estimation of the failure probability for the severity level S using the MCS method. 


! The exceedance probability represents a probability of exceeding a certain limit that can be 
quantified and demonstrated to be less than an acceptable value. 
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Figure 7.3 shows the failure probability of the severity level S? using the MCS 
method with a number of samples Nme = 1800. The number of Hits is 335 and 
the number of Misses is 1465. Considering the graph for severity level S? as 
shown in figure 7.3, here, the condition seems to be fulfilled for more numbers 
of samples. This means that the number of Hit samples is increased and also 
the value of uS l! [km/h] for which the condition seems to be fulfilled. After the 
first few samplings, this failure probability value falls and then becomes stable 
at a level Pe(S2?|C}) = a = 0.19. In addition, the standard error becomes 
stable gradually at the value ep (SICH) = 0.0092 to be less than the required 
standard error 65, with 0.05. Ultimately, the standard error does not seem to be 
entirely nullified, but is really low. It makes a cloud of the Hit samples slightly 
shifted towards the right side. It is important to note that the figure 7.3 shows 
the ratio of Hits and their corresponding velocity ye" [km/h]. It is obvious that 


as the velocity ue ! [km/h] increases, the sensor error dêl le] also increases for 


the specified criticality range S. 
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Figure 7.3: Estimation of the failure probability for the severity level 82 using the MCS method. 
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7 Reliability Analysis Using Sampling Methods 


Figure 7.4 illustrates the failure probability of the severity level S! using the 
MCS method, with a number of samples Nnc= 500. The number of Hits is 
257 and the number of Misses is 243. Here, the Hit condition is improved as 
compared to the previous severity level. For the severity level S}, the values 
of v& 'Tkm/h] have also increased and the cloud of Hit samples moves right- 
wards again. According to the results of this sampling, the failure probability 
of this level of severity is high at the beginning of sampling. It falls imme- 
diately after some samples and then stabilizes at a certain level of P;(S!|C!) 
= Ar = 0.51 with infinitesimal difference in each sampled value. The standard 
error ep, (S!|C!) gives the value 0.022 as abort criterion of the simulation, 
which is smaller than the target standard error ôp,. 
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Figure 7.4: Estimation of the failure probability for the severity level S! using the MCS method. 


Figure 7.5 describes the failure probability for the severity level S using the 
MCS with a number of samples Nme = 1600. The number of Hits is 321 and the 
number of Misses is 1279. This sampling does show the stability in the failure 
probability with P;(S°|C!)= = = 0.2. The value of the standard error is less 
than the previous case with ep, (Se IC}) = 0.01. 
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7.3 Adaptive Importance Sampling 


Finally switching to the figure 7.5, the cloud of Hit samples lies to the right 
side of the graph for the severity level S°. Here, the target Hits are a little less in 
number compared to severity levels S} and S?. It is significant to note, that with 
correspondence to figure 7.1 the studied range contains only a small number of 
events. Thus, rendering this statistic improbable. However, the indication from 
the graph follow accordingly to the expected flow. 
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Figure 7.5: Estimation of the failure probability for the severity level S? using the MCS method. 


The figures 7.2 to 7.5 illustrate that there is a movement in the position of the 
target Hits starting from the bottom left moving towards the top right through 
the severity levels, looking like a cloud of Hits. 


7.3 Adaptive Importance Sampling 


Since the estimator variance corresponds to the confidence, variance reduction 
techniques aim at influencing the sampling in such a way that the estimator 
variance becomes smaller. 
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7 Reliability Analysis Using Sampling Methods 


The Importance Sampling is a variance reduction method to guide the sampling 
using biased normal distribution for variance reduction. The estimator includes 
the Importance Sampling weight to warrant unbiasedness of the estimator. The 
samples are not generated following the prescribed joint PDF of fz (z), but with 
an importance sampling density denoted as hy(y). Setting hy(y)/hy(y) into 
the probability integral does not change its value, but the integral provides the 
expected value. 
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The AIS reduces the estimator variance and involves several simulation 
runs [O*18]. The importance sampling density hy(y) may have a larg- 
er scatter. In the first iteration, the samples falling into the failure domain 
{z € R"g(z) < y} :Z|g(z) < y are statistically evaluated. The result is used 
to define the parameter of normal Gaussian distribution type for the subse- 
quent Importance Sampling iterations. To increase the number of Hits, the AIS 
method is managed through the use of information about the LSF domain. Each 
sample is weighted according to the ratio of the original density function fz(z) 
to the importance density function hy(y) to ensure correct statistics [KS13]. 
Furthermore, the AIS techniques can be applied to efficiently search for critical 
scenarios by speeding up the parameter space exploration [O*18]. Equations 
7.8 and 7.9 show the expected values of the importance sampling density Ay (y) 
according to the estimated mean and covariance of the samples in the failure 
domain in iterative steps [Li07]. 


E[Y] = E[Z|g(z) < y] (7.8) 


E[YY"] =EIZZ’|g(z) < 7] (7.9) 
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7.3 Adaptive Importance Sampling 


The third iteration should be performed to prove the stability of the results in an 
iterative adaption scheme. Accordingly, the goal of the importance sampling 
density hy(y) is to converge the estimator standard error to zero by adjusting 
the importance sampling density hy(y) iteratively [BMC15]. Typically, the 
target standard error for the termination criteria is set to 0.1, but in order to 
conduct more samples, the target standard error was set to 0.05. (The value 
= 0.05 was previously utilized in MCS sampling). 


Figure 7.6 shows the estimation of failure probability of the severity level 
S? using the AIS method with number of samples Nas = 800. The AIS algo- 
rithm applies four iterations, where each iteration consists of 200 samples. The 
number of Hits is 512 and the number of Misses is 288. The standard error 
is calculated to be ep, (S IC!) = 0.0088. The standard error of the estimator 
doesn’t appear to be entirely eliminated, but is extremely low. The failure 
probability is computed to be Pr(S3|C}) = 0.1 as confirmation of the MCS 
result for the same severity level. 
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Figure 7.6: Estimation of failure probability for the severity level SÌ using the AIS method. For 
comparative purposes, these calculations in MCS are shown in figure 7.2. 
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7 Reliability Analysis Using Sampling Methods 


The figure 7.7 describes the behavior of the failure probability and standard 
error after sampling using number of samples Nas = 900. The sampling is 
performed in five iterations, with the first three iterations having a budget of 
100 samples, the fourth with 200 samples, and the fifth with 400 samples. 
The number of Hits is 434 and the number of Misses is 446. The standard 
error is computed to be ep, (S2ICH) = 0.0088. This shows that with AIS, 
the variance is successfully eliminated. This proves that the AIS method is 
effective in eliminating the variance with less numbers of samples compared 
to the MCS method. The failure probability is calculated to be P;(S?|C!) 
= 0.18 as confirmation of the MCS result for the severity level S?. 
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Figure 7.7: Estimation of failure probability for the severity level S? using the AIS method. For 
comparative purposes, these calculations in MCS are shown in figure 7.3. 


As shown in figure 7.8, the probability of failure P(S} |C}) with 0.52 using the 
AIS method confirms the result extracted from the MCS method for the same 
severity level. The convergence of the AIS method occurs with the required 
standard error of 0.05 using a number of samples Nas = 800. Accordingly, the 
standard error ep,(S !Ic}) of 0.036 is used as abort criteria for the simulation 
runs. 
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7.3 Adaptive Importance Sampling 


The sampling is performed in four iterations, with the first three iterations 
having a budget of 100 samples and the fourth with 500 samples. The number 
of Hits is 616 and the number of Misses is 184. 
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Figure 7.8: Estimation of failure probability for the severity level S! using the AIS method. For 
comparative purposes, these calculations in MCS are shown in figure 7.4. 


Figure 7.9 shows that the failure probability at severity level S? using the AIS 
method and number of samples Nas = 800. The sampling is executed in four 
iterations, with the first three iterations having a budget of 100 samples and the 
fourth with 500 samples. The number of Hits is 581 and the number of Misses 
is 219. Because of the very few available TO events of the cluster C at the 
severity level S? as shown in figure 7.1, the AIS method generates a statistical 
uncertainty with the result of the failure probability. As aforementioned, due to 
poor sampling in the region of TIC > 2[s], there were not enough sample 
to explorate the values within the specific used budget of samples, resulting in 
an non-coherent probability failure not achieving ep, (S°IC!) less than 0.05. 
Figure 7.9 shows that AIS method is the least suitable for these error ranges of 
TE and MCS method ought to be used instead. 
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7 Reliability Analysis Using Sampling Methods 
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Figure 7.9: Estimation of failure probability for the severity level s? using the AIS method. For 
comparative purposes, these calculations in MCS are shown in figure 7.5. 


Figures 7.6 to 7.9 show a similar type of movement of the Hit cloud from 
the bottom left to the top right in comparison with figures 7.2 to 7.5. In 
many cases, the AIS method requires fewer samples to achieve the required 
standard error. For several iterations, the parameters of the importance sampling 
density function hy (y) can be determined adaptively. The probability of failure 
decreases from S} to S, while S? occurs with a bigger probability than S!. 
The safety statement can be obtained in that the failure probability for each 
traffic scenario is approximated by estimating the exceedance region in the 
parameter space. At the beginning of the AIS method, the required standard 
error dp, of 0.1 is chosen for the investigations, which generally represents a 
good confidence in the result. While the AIS fulfills its required standard error 
of 0.1 after a few simulation runs, but the iteration results differ strongly, the 
required standard error is reduced to dp, = 0.05 to force more iterations for the 
AIS method. For comparability reasons, 6p, = 0.05 is also given for the MCS 
algorithm in all severity levels. The MCS method is used as a reference, where 
the results of MCS and AIS methods confirm each other. 
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8 Conclusions 


The incomplete and implicit requirements are a part of the verification 
challenges of ADFs. There is no complete public computer-aided traffic 
simulator that contains the rules for dealing with exceptions. Moreover, testing 
can detect the presence of errors, but not their absence. In addition, the Pesticide 
paradox phenomenon occurs after fixing the software errors found by a scenario 
catalog. As a consequence, the same catalog may no longer be suitable for the 
residual errors. The incompleteness of the functional requirements can thus 
be covered by requirements extraction from the field observation. The on-road 
testing plays a decisive role to identify the missing functional requirements and 
transform these requirements adaptively into the deployed scenario catalogs. 
The scenario-based testing assumes that the majority of the mileage on the 
motorway is accomplished without particular events, while critical scenarios 
in real traffic are rather rare and randomly distributed. The identification and 
reproducibility of critical scenarios from the KDD of on-road testing in lab- 
oratory simulations or proving grounds leads to a significant reduction of the 
FOT distances required for statistical approval. The scenario-based functional 
decomposition, therefore, splits complex functions into sub-functions in order 
to identify relevant driving scenarios that reduce the approval effort for ADFs. 


8.1 Executive Summary 


The status quo evaluation refers to large-scale verification as one of the decisive 
challenges for the economical, reliable and safe use of ADFs in CMV product 
engineering. Therefore, the systems engineering has established data-based 
and knowledge-driven test methods to assure the required dependability of 
their products. 
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8 Conclusions 


The required test distance is approximately 220 million kilometers with error- 
free driving to show that the automated CMV operates with a confidence level 
of 95% as safely as a human driver, as indicated in section 2.4. 


RQ1: How can the knowledge-based and data-driven test platforms be 
combined in a complementary and collaborative manner? 


Due the rare nature of critical events, the FOTs will require a prohibitive 
number of driving hours to prove the statistical superiority. At the same time, 
the required total mileage is increased if an error occurs during the validation 
process. Therefore, the sole reliance on FOTs is inadequate and, in particular, 
time and cost-intensive when applied to the next generation of ADFs, e.g. 
truck platooning and hub-to-hub transportation. Owing to the complexity of 
the ADFs, a test oracle for safety certification is unlikely to be provided by a 
single test platform. Therefore, innovative approaches are required to enable 
systematic testing under reproducible conditions with regard to robustness, 
reliability and safety, as discussed in chapters 3 and 5. In addition, functional 
decomposition is also necessary to support the argumentation for a reasonably 
low residual risk resulting from imperfections of the environmental perception 
sensors, as elucidated in chapter 4. 


RQ2: How can the ADFs be effectively and efficiently tested? 


The context-driven approach — as indicated in section 3.2 — employs a 
grey-box test strategy that combines the insights from the observed road data 
with functional requirements within the ODD coverage. In figure 3.4, the 
argument structure of the context-driven test concept is shown with the help 
of the GSN. The required test termination criteria can be met by quantify- 
ing the conflicts between the test concept requirements, and thus, achieving a 
trade-off between efficiency and effectiveness criteria of the test procedures. 
The correlation between the various test coverage criteria intends to support 
controlling the decision of choosing whether more test kilometers need to 
be collected or more simulations require to be carried out. The optimized test 
strategy requires a selection of the necessary test methods for various scenarios 
and their interaction with other test methods. Thereby, new and innovative 
approaches need to be designed, especially in simulation and in the laboratory. 
The knowledge-based test platforms are executed at component and system 
levels to generate the corresponding test evidence, as explained in chapter 4. 
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8.1 Executive Summary 


Moreover, the test results of the data-driven test platforms are evaluated using 
statistical extrapolation techniques and field-based observations within the 
ODD coverage, as discussed in chapter 5. 


RQ3: How can the prospective risk be measured in order to achieve rea- 
sonable termination criteria for the testing of the ADFs? 


As indicated in chapter 5, the measurable safety approach envisages the identi- 
fication of failure types to break down the functional complexity. Furthermore, 
the presented framework structure utilizes a back-end database filled with 
catalogs of corner driving scenarios from different sources of field-based ob- 
servations. The extensive data-sets are collected world-wide under realistic 
driving conditions with test CMV fleets. The evaluation of algorithms requires 
an efficient search and interpretation of relevant traffic situations with help of 
RCA to modify the algorithm accordingly. In this scheme, the processing chain 
includes clustering of multivariate time-series data-sets and finding critical 
driving situations to identify and allocate the necessary test cases for various 
suitable test environments. The platform-independent mechanism is intended 
to offer a consistent scenario description format for the various test envi- 
ronments. Further, these new test cases complement the existing test cases 
developed from expert knowledge in an adaptive ODD coverage manner. 


A case study is provided to illustrate the measurable safety framework, in 
which the behavior of CMVs equipped with an AEB system with respect to 
stationary objects is discussed. The AEB can also unexpectedly issue warnings 
and brake the vehicle if it detects stationary objects next to the vehicle’s own 
lane, e.g. broken-down vehicles, signs, bridges and traffic islands. The extracted 
clusters and their parameter space define the probability distributions of the 
associated parameters of each logical scenario. The minimum TTC is used 
as a prototypical safety measure. Thereby, the sensitivity analysis reduces the 
number of input variables to the most important ones — as discussed in chapter 
6 — , which influence the safety measure. Subsequently, the reliability analysis 
identifies the failure region in the parameter space to predict the probability of 
failure for each logical scenario using MCS and AIS methods, as indicated in 
chapter 7. The generated tolerable risk curve is then used as a reference safety 
threshold — as portrayed in figure 6.1 — for the test termination criteria. 
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8 Conclusions 


Meanwhilst, the real PoS can only be provided after the market introduction of 
automated CMVs in which the probability of collisions can be estimated based 
on the relevant parameter space within the ODD. However, the probability 
of failure can be approximated by estimating the exceedance probability of 
parameters with higher criticality while validating the performance of existing 
levels of automation. Accordingly, there is a hypothesis based on the assump- 
tion that erroneous triggering events of the present ADFs during the field 
observation can be considered as disengagement events for automated CMVs. 
Therefore, the prospective risk assessment of an existing ADF provides refer- 
ence values for the risk acceptance threshold. Such reference values act as a 
benchmark for the further development and optimization of a similar ADF at 
the next level of automation. 


8.2 Technical and Scientific Contributions 


The proposed framework enables functional verification of ADFs on the 
embedded ECUs in complex automotive sensor networks. The framework 
utilizes a real-time capable system architecture in a distributed heteroge- 
neous co-simulation environment. In this architecture, object-list-based sensor 
models are designed to simulate realistic sensor behavior. The real-time 
interaction between the HiL test bench and the DuT imposes timing constraints 
to ensure traceability and reproducibility of the test results. The described 
architecture contains both the structural design and time-efficient integration 
into the test bench and is applied to RADAR and camera sensors. An impor- 
tant feature of the architecture is the high portability between various driving 
simulation frameworks due to well-defined in-/output interfaces [BMK* 16]. 
Furthermore, a run-time fault injection has been introduced to simulate sensor 
failures, such as latency, detection failure and false one-to-many object labeling. 
These failures are used for testing the robustness of the ADFs. Accordingly, a 
phenomenological sensor model can be developed iteratively. Sensor failures 
can be added in a continuous manner in order to achieve increasing degrees 
of realism. For efficient verification, an astute selection of relevant test sce- 
narios is conducted to avoid repetitive test scenarios. The research results of 
this thesis have been published in internationally well-recognized journals and 
peer-reviewed conferences. 
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8.2 Technical and Scientific Contributions 


Six invention disclosures focusing on test procedures have been filed and used 
as references in this work. In the DE 10201700997 1Al, the invention relates 
to a method for testing a lane assistance system for a vehicle by extracting 
an ontology-based category of adequate and relevant scenarios for existing 
field tests [ESF19]. In the DE 102018004429A1, the invention deals with 
a cluster analytical characterization of driving situations based on detected 
sensor signals for the detection of surroundings and their system reactions in 
the driving operation of the vehicle [ESS*19c]. In the DE 102018005864A1, 
the invention relates to a method for testing a blind spot assistance system for 
a vehicle, particularly for a truck. On the basis of the Tractrix zone of the Ego- 
vehicle, objects are detected by sensors and their distance and relative speed 
to the Ego-vehicle are measured. The cluster analysis is applied to determine 
the criticality of the detected ambient objects [ESS* 19b]. 


In the DE 102018005865A1, the invention relates to a method for testing a 
map-based system for speed limit in a vehicle. The system detects traffic signs 
based on maps and fused image, wherein acquired information to be tested are 
classified by means of a cluster analysis for scenarios into equivalence classes 
[ESS*19a]. In the DE 102020005507A 1, the invention refers to a method for 
testing an emergency braking function of a vehicle based on a confidence for 
existing field tests. Thereby, the sensitivity and reliability analysis identifies a 
failure range in the parameter space to predict a failure probability for each 
extracted logical scenario using sampling methods [ESD20]. 


In the DE 102020006644A 1, the invention relates to a method for evaluating 
systematic and statistical errors of multi-radar-based recognition systems of a 
vehicle. According to the invention, exceptions from field-based observations 
and knowledge experience are identified by means of processes of knowledge 
development by clustering acquired data records and determining the excep- 
tions in order to identify and assign necessary test data records for various 
suitable test environments. By means of the determined test data-sets, test 
data-sets developed on the basis of expert knowledge are completed in an 
adaptive test coverage [EDA20]. 
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8 Conclusions 


8.3 Future Research Directions 


The presented work raises several issues that require substantial future research 
activities. The quality of ADFs depends primarily on the environmental sensors 
providing the vehicle’s environmental perception as the basis for situation 
analysis and decision making. An acceptable level of maturity of these sensors 
must be accomplished as a prerequisite for an adequate field validation. The 
logical and statistical errors or functional deficiencies are assessed by objective 
and subjective evaluation criteria. 


Since it is not possible to guarantee complete safety for automated CMVs, one 
of the major challenges in automated truck driving is to argue for a reasonably 
low residual risk resulting from limitations of the environmental perception 
sensor. Currently, relevant safety norms do not support such arguments. While 
the Threat Assessment and Remediation Analysis (TARA) is one of the key 
activities defined in the ISO/SAE 21434 for automotive cyber-security, a SOTIF 
similar standard is also required for security issues. 


Moreover, the standards for safety (ISO 26262 and ISO/PAS 21448) and se- 
curity (ISO/SAE 21434) need to be more harmonized. The cyber-security is a 
condition in which assets are adequately protected against threat scenarios to 
electrical or electronic components of road vehicles and their functions [fS*20]. 
Therefore, further research will include the application of the test method for 
security issues. These activities have to be integrated into an ASE approach 
that supports the structure of the context-driven test concept. This research 
work needs to be complemented by activities with standard organizations to 
form a consensus on risk evaluation and acceptable argumentation structures 
that would feed into future standards and CoP guidelines for the safeguarding 
of ADFs. 
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9 Glossary 


9.1 List of Acronyms 


ABA 
ACC 
ACSF 
ADA 
ADAS 
ADASIS 
ADF 
AEB 
ANSI 
ASE 
ASIL 
ASPICE 


AST 
ATA 
BMWi 
CAN 
CFAR 
CMMI 
CMV 
CoP 
DBSCAN 
DDT 


Preliminaries 


Active Brake Assist 

Adaptive Cruise Control 
Automatically Commanded Steering Function 
Active Drive Assist 

Advanced Driver Assistance System 
ADAS Interface Specification 
Automated Driving Function 
Autonomous Emergency Braking 
American National Standards Institute 
Automotive Systems Engineering 
Automotive Safety Integrity Level 


Automotive Software Process Improvement and Capability 
dEtermination 


Adaptive Stress Testing 

American Trucking Associations 

German Federal Ministry for Economic Affairs and Energy 
Controller Area Network 

Constant False Alarm Rate 

Capability Maturity Model Integration 

Commercial Motor Vehicle 

Code of Practice 

Density-Based Spatial Clustering of Applications with Noise 
Dynamic Driving Task 
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9 Glossary 


DESTATIS 
DIN 

DIS 

EC 

ECU 

E/E 
eHorizon 
FMI 

FN 

FOT 
FOV 

FP 

GPS 
GSN 
HARA 
HD 

HiL 

HoL 


LiDAR 
MC/DC 
MIT 

MiL 
MOBATSim 
MOP 

MPP 
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Federal STATIStical Office of Germany 
German Insitute for Standardization 

Draft International Standard 

European Commission 

Electronic Control Unit 

Electrical and/or Electronic 

electronic Horizon 

Functional Mock-up Interface 

False Negative 

Field Operational Test 

Field of View 

False Positive 

Global Positioning System 

Goal Structuring Notation 

Hazard Analysis and Risk Assessment 

High Definition 

Hardware-in-the-Loop 
Hardware-in-the-open-Loop 

Hardware/ Software 

International Electrotechnical Commission 
Institute of Electrical and Electronics Engineers 
International Organization for Standardization 
Key Performance Indicator 

Lane Departure Protection 

Lane Departure Warning 

Light Detection And Ranging 

Modified Condition/Decision Coverage 
Ministry of Industry and Information Technology 
Model-in-the-Loop 

MOdel-Based Autonomous Traffic Simulation Framework 
Metamodel of Optimal Prognosis 

Most Probable Path 


9.1 List of Acronyms 


MRC Minimal Risk Condition 

MTBF Mean Time Between Failures 

MVP Minimum Viable Product 

NCAP New Car Assessment Programme 
NDS Navigation Data Standard 

NHTSA National Highway Traffic Safety Administration 
NL Natural Language 

ODD Operational Design Domain 

OEDR Object and Event Detection and Response 
PAS Publicly Available Specification 
PCA Principal Component Analysis 

PoS Proof of Safety 

PPV Positive Predictive Value 

QG Quality Gate 

QM Quality Management 

RADAR RAdio Detection And Ranging 
RCA Root Cause Analysis 

RCS Radar Cross Section 

ROC Receiver Operating Characteristic 
SAE Society of Automotive Engineers 
SDLC Software Development Life Cycle 
SWE Software Engineering Process Group 
SENSORIS SENSOR Interface Specification 
SGA Side Guard Assist 

SiL Software-in-the-Loop 

SOTIF Safety Of the Intended Functionality 
SPI Safety Performance Indicator 

STL Signal Temporal Logic 

SVM Support Vector Machine 

TaaS Transport-as-a-Service 

TN True Negative 

TP True Positive 
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9 Glossary 


TPEG 
TRL 
TSR 
TIG 
TTR 
UL 
UML 
UN/ECE 
UNGA 
UNR 
U.S. 
USDOT 
VCRT 
VRU 
VSSA 
V&V 
VVA 
WHO 
XiL 


ABox 
ADDR 
ADTF 
AWS 
CCR 
CRF 
CSV 
DL 
DuT 
DVI 
EDR 
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Transport Protocol Experts Group 
Technology Readiness Level 

Traffic Sign Recognition 

Time To Collision 

Time To React 

Underwriters Laboratories 

Unified Modeling Language 

United Nations Economic Commission for Europe 
United Nations General Assembly 

United Nations Regulation 

United States 

United States Department of Transportation 


Vienna Convention on Road Traffic 
Vulnerable Road User 

Voluntary Safety Self Assessment 
Verification and Validation 

Verification, Validation and Accreditation 
World Health Organization 
X(something)-in-the-Loop 


Framework Conception 


Assertional Box 

Automated Driving Data Recorder 
Automotive Data and Time-triggered Framework 
Amazon Web Services 

California Code of Regulation 
Camera Reference Frame 
Comma-Separated Values 
Description Logic 

Device under Test 

Digital Visual Interface 

Event Data Recorder 


9.1 List of Acronyms 


REST-API 


RTT 
RTOS 
TBox 
TCP 
UDP 
V2X 
WGS84 
XCP 
XML 


German In-Depth Accident Study 
Global Navigation Satellite System 
Hierarchical Agglomerative Clustering 
Hadoop Distributed File System 
highway Drone data-set 
Human-Machine Interface 

Knowledge Discovery in Databases 
Inertial Measurement Unit 
intersection Drone data-set 


Karsruhe Institute of Technology and Toyota technological 
Institute data-set 


Long Term Evolution 

Measurement Data Format 

Network Attached Storage 

Normalized Cross Correlation 

National Motor Vehicle Crash Causation Survey 
Over-The-Air 

Ontology Web Language 

Plug On Device 

Precision Time Protocol 

Resource Description Framework 


REpresentational State Transfer - Application Programming 
Interface 


Round Trip Time 

Real Time Operating System 

Terminological Box 

Transmission Control Protocol 

User Datagram Protocol 

Vehicle to X(everything) 

World Geodetic System 1984 

Universal (X) Measurement and Calibration Protocol 
eXtensible Markup Language 
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9 Glossary 


AIS 
ANN 
CDF 
DoE 
LSF 
MCS 
MLS 
MOP 
PDF 
PPMCC 
RDE 
SVR 
SWRL 
TARA 
THW 


Framework Implementation 


Adaptive Importance Sampling 

Artifical Neural Network 

Cumulative Distribution Function 

Design of Experiments 

Limit State Function 

Monte Carlo Sampling 

Moving Least Squares 

Metamodel of Optimal Prognosis 

Probability Distribution Function 

Pearson Product-Moment Correlation Coefficient 
Real Driving Emissions 

Support Vector Regression 

Semantic Web Rule Language 

Threat Assessment and Remediation Analysis 
Time Head Way 
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Preliminaries 


longitundinal acceleration of the Ego-vehicle. 

confidence level of the Binomial distribution. 

accumulated driving distance for the statistical PoS. 
falsified relative longitudinal distance of the preceding pedestrian. 
falsified relative lateral distance of the preceding pedestrian. 
relative lateral distance to the left lane marking. 

relative lateral distance to the right lane marking. 

cut-out distance for lane change of the following vehicle. 
wheelbase of the Ego-vehicle tractor. 

index of the hazardous driving situations. 

set of the hazardous driving situations. 


9.2 List of Variables and Constants 


7 
4 


Tollide 


tco 


index of the number of detected objects. 
index of the number of tracked objects. 
index of the sensor node. 


number of the hazardous events along the cumulative driving 
distance for the statistical PoS. 


number of the sensor nodes for environmental perception within 
the vehicle-mounted sensor setup module. 


failure rate of the hazardous events. 
failure rate of an ADF without driver supervision. 
benchmark failure rate of a human driver. 


fatality factor of the ADF without driver supervision in relation to 
the benchmark failure rate of a human driver. 


reliability as a complementary of the failure rate A. 


number of the hazardous events observed by field operational 
tests. 


required mode of rolling within an ACC function. 
cut-out delay time for lane change of the following vehicle. 
longitudinal velocity of the Ego-vehicle. 


longitudinal velocity of the object with the identification number 
I: 


longitudinal velocity of the preceding object with the 
identification number 2. 


relative lateral velocity to the left lane marking. 
relative lateral velocity to the right lane marking. 


distance from crossing point at the origin to a horizontal 
extremity of a Lemniscate elliptic function. 


Framework Conception 


lateral acceleration of the Ego-vehicle. 


longitudinal acceleration between the Ego-vehicle and the object 
ahead. 


lateral acceleration between the Ego-vehicle and the object ahead. 
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arl vertical acceleration between the Ego-vehicle and the object 
ahead. 

q=! longitudinal distance between the Ego-vehicle and the object 
ahead. 

a lateral distance between the Ego-vehicle and the object ahead. 

ae vertical distance between the Ego-vehicle and the object ahead. 

d longitudinal distance between the Ego-vehicle and the stationary 
vehicle in front, which is detected by the camera sensor model. 

dy longitudinal distance between the Ego-vehicle and the stationary 
object in front, which is detected by the RADAR sensor model. 

fa sampling rate of the VxWorks® RTOS-based HiL simulator. 

Im sampling rate of the 3D simulation environment. 

F; weight force on the rear axle. 

Kego RADAR-based estimation of curvature for the Ego-vehicle 
trajectory. 

Mb braking torque of the Ego-vehicle. 

Py foot placement on the brake pedal. 

r proximity distance index between two time series vectors. 

ts sampling period of the vehicle dynamics model. 

to turn-around time. 

U number of subsets for features of sensor modelling. 

u longitudinal velocity of the Ego-vehicle. 

u longitudinal velocity between the Ego-vehicle and the object 
ahead. 

d lateral velocity between the Ego-vehicle and the object ahead. 
u vertical velocity between the Ego-vehicle and the object ahead. 
x longitudinal position of the Ego-vehicle in Cartesian 

co-ordinates. 
y lateral position of the Ego-vehicle in Cartesian co-ordinates. 
Z vertical position of the Ego-vehicle in Cartesian co-ordinates. 
$ roll angle of the Ego-vehicle. 
0 pitch angle of the Ego-vehicle. 
u heading (yaw) angle of the Ego-vehicle. 
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max 
c 


Ob 


qê! 
x 
él 

dy 


él 
dite 


min 
dy 

max 
dy 


ep, (SZIC.) 
ep, (SFICH) 
ep, (S; ICS) 
ep, (SICH) 
Kroad 


Ki 


ke! 


Pr(S3 IC.) 


maximum roll angle with zero weight force acting on the rear 
axle. 


pitch angle while braking in the straight-ahead direction. 


Framework Implementation 


longitudinal distance between the Ego-vehicle and the object 
ahead at the time of triggering the first escalation event. 


lateral distance between the Ego-vehicle and the object ahead at 
the time of triggering the first escalation event. 


distance between the TTC of FOT and the theoretical ideal TTC 
curve from HiL testing. 


lower bound of the truncated normal distribution of the lateral 
distance between the Ego-vehicle and the object ahead at the time 
of triggering the first escalation event. 


upper bound of the truncation normal distribution of the lateral 
distance between the Ego-vehicle and the object ahead at the time 
of triggering the first escalation event. 


standard error of SẸ of the cluster C} at the escalation level E£. 
standard error of S? of the cluster C} at the escalation level &l. 
standard error of S} of the cluster C} at the escalation level &l. 
standard error of S? of the cluster C} at the escalation level &l. 
actual road curvature. 

road curvature at the position (i) of the clothoid. 


curvature of the Ego-vehicle at the time of triggering the first 
escalation event. 


lower bound of the truncated normal distribution for the road 
curvature at the time of triggering the first escalation event. 


upper bound of the truncated normal distribution for the road 
curvature at the time of triggering the first escalation event. 


length of the clothoid arc to the point (i). 


failure probability of S of the cluster C} at the escalation level 
ey 


175 


9 Glossary 


Pr(SzICH) 
Pr(S, ICH) 
Pr(SSICH) 


Te! 


ref 


TTCÊ! 


fot 
TES! 


min 


vë! 


failure probability of S? of the cluster C; at the escalation level 
8l. 

failure probability of S} of the cluster C! at the escalation level 
él. 

failure probability of S? of the cluster C! at the escalation level 
El. 

Time to Collision from the HiL testing. 

Time to Collision from the on-road testing. 

minimum Time to Collision at the last moment when the object is 
classified as relevant. 


longitudinal velocity of the Ego-vehicle at the time of triggering 
the first escalation event. 


lower bound of the triangular distribution of the longitudinal 
velocity at the time of triggering the first escalation event. 

upper bound of the triangular distribution of the longitudinal 
velocity at the time of triggering the first escalation event. 

mode of the triangular distribution of the longitudinal velocity at 
the time of triggering the first escalation event. 

longitudinal position at the position (i) of the clothoid. 

lateral position at the position (i) of the clothoid. 


direction angle between the tangent of the current position (i) of 
the clothoid and the initial direction of the straight line. 
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set of missing specifications for the intended behavior within the 
three-circle Venn diagram. 

set of robust unspecified behaviors within the three-circle Venn 
diagram. 

intersection of the three Venn diagram sets of the specified, 
implemented and intended behaviors. 

Venn diagram set of the missing implementations for correct 
specifications. 


9.3 List of Quantities and States 


VMiL 


set of unexpected wrong behaviors within the three-circle Venn 
diagram. 

set of wrong specifications or technical limitations within the 
three-circle Venn diagram. 


set of missing implementations for wrong specifications within 
the three-circle Venn diagram. 

set of implemented behaviors within the three-circle Venn 
diagram. 

set of intended behaviors within the three-circle Venn diagram. 
set of specified behaviors within the three-circle Venn diagram. 
first escalation event, where visual and acoustic alarms are issued 
to warn the driver about a possible collision with a stationary 
object. 

second escalation event, where haptic and automatic particle 
braking is applied to assist the driver in braking in order to 
prevent a collision with a stationary object. 

third escalation event, where emergency braking occurs at a point 
where the collision with a stationary object is imminent. 

first position of the stationary object within the Tractrix area of 
the Ego-vehicle defined in the Cartesian co-ordinates. 

second position of the stationary object within the Tractrix area 
of the Ego-vehicle defined in the Cartesian co-ordinates. 

third position of the stationary object within the Tractrix area of 
the Ego-vehicle defined in the Cartesian co-ordinates. 

fourth position of the stationary object within the Tractrix area of 
the Ego-vehicle defined in the Cartesian co-ordinates. 

fifth position of the stationary object within the Tractrix area of 
the Ego-vehicle defined in the Cartesian co-ordinates. 

recorded sequence during on-road testing by reporting an event 
of measurement and detection errors. 

newly produced sequence after updating of the ECU software 
with bug fixing of the faulty object detection during regression 
and progression tests. 

model result of a back-to-back test within a MiL test setup. 
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VSiL 


oI 


oo! 


generated code result of a back-to-back test within a SiL test 
setup. 


Framework Conception 


mean of the time-series vector of the variable A. 

mean of the time-series vector of the variable B. 

standard deviation of the time-series vector of the variable A. 
standard deviation of the time-series vector of the variable B. 
group of driving situations with a combined curve to the left. 
group of driving situations with a S-shaped clothoid to the right. 
group of driving situations with a S-shaped clothoid to the left. 
group of driving situations with a combined curve to the right. 


group of driving situations with a deviation to the detected right 
lane marking and a return afterwards from this deviation. 


group of driving situations with a sudden jump in the distance to 
the lane, which indicates the detection of new lane lines. 


group of driving situations with a deviation and a temporary 
sudden change of the distance to the right line and back to normal 
deviations due to painted islands. 


distance measure of the NCC algorithm. 

distance measure of the NCC algorithm with a time shift sj. 
HAC function. 

resulting cluster of the HAC algorithm. 

merged cluster according to the complete-linkage criterion. 
merged cluster according to the complete-linkage criterion. 
radar sensor model. 

3D cone model of a perception sensor. 

camera sensor model. 

first relevant object detected by the camera sensor model. 


second relevant object detected by the camera sensor model. 
first relevant object detected by the radar sensor model. 


second relevant object detected by the radar sensor model. 
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timestamp message of the camera sensor model. 
timestamp message of the fusion module. 
timestamp message of the radar sensor model. 
point of the normalized rolling standard deviation. 
rolling window of the NCC algorithm. 

input of discrete state. 

input update of discrete state. 

input of discrete state. 

output of continuous state. 


Framework Implementation 


expectation of a target indicator function. 

standard error of the failure probability estimator. 
joint density function for the random vector Z. 
indicator function. 

importance sampling density. 

limit state function. 

number of samples according to the MCS method. 
number of samples according to the AIS method. 
probability of failure. 

estimator of the failure probability. 


prbability of occurrence for the group of driving situations with a 
combined curve to the left. 


overall probability of occurrence for a functional scenario. 
variance of the failure probability estimator. 
coefficient of determination. 


prospective risk assessment for the group of driving situations 
with a combined curve to the left. 


hypothesis for low severity. 
hypothesis for medium severity. 
hypothesis for high severity. 
hypothesis for critical severity. 
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y predefined safety margin. 

6p, required standard error for the failure probability estimator. 
H mean of the parent normal distribution. 

Hrrcë! mean of the variable TICH,, 

Hrrcê! mean of the variable Mo, 

o standard deviation of the parent normal distribution. 
Irrcäı standard deviation of the variable Tice. 

Orres! standard deviation of the variable TIC... 

Q PDF of the normal Gaussian distribution. 

® CDF of the normal Gaussian distribution. 


N 


realization of Z in the variable space. 
Z random vector. 


9.4 List of Matrices and Vectors 


Preliminaries 


On controllability of hazardous events. 

OY controllable hazardous event in general. 

© simply controllable event, where 99% or more of all drivers or 
other traffic participants are usually able to avoid hazardous event. 

O? normally controllable event, where 90% or more of all drivers or 
other traffic participants are usually to avoid hazardous event. 

(ON difficult to control or uncontrollable event, where less than 90% 


of all drivers or other traffic participants are usually able, or 
barely able, to avoid the hazardous event. 


pk) set of sensor properties at detection level for each sensor node. 

a) set of sensor properties at object level for each sensor node. 

p subset of the sensor properties at detection level for each sensor 
node. 

a subset of the sensor properties at object level for each sensor 
node. 

R scenario-based risk assessment of ADFs. 
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number of sensor nodes for environmental perception within the 
vehicle-mounted sensor setup module. 


severity of the hazardous events. 

no injuries. 

light and moderate injuries. 

severe and life-threatening injuries (survival probable). 
life-threatening injuries (survival uncertain or fatal injuries). 
exposure of the hazardous events. 

probability of unlikely occurrence of hazardous events. 

very low probability of hazardous events. 

low probability of hazardous events. 

medium probability of hazardous events. 


high probability of hazardous events. 


Framework Conception 


time series vector of the variable A. 

time series vector of the variable B. 

overall set of the implemented features of a sensor model. 
subset of the implemented features of a sensor model. 


set of configuration parameters for each subset of the 
implemented features of a sensor model. 


set of detection-level sensor properties for a sensor model. 

set of object-level sensor properties for a sensor model. 

rotation matrix from Ego-vehicle co-ordinate system to inertial 
co-ordinate system. 

rotation matrix from sensor co-ordinate system to Ego-vehicle 
co-ordinate system. 

rotation matrix from inertial co-ordinate system to sensor 
co-ordinate system. 

translation vector from Ego-vehicle to initial co-ordinate system. 
translation vector from sensor co-ordinate system to Ego-vehicle 
co-ordinate system. 
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translation vector from object co-ordinate system to initial 
co-ordinate system. 


roto-translation matrix from initial co-ordinate system to sensor 
co-ordinate system. 


roto-translation matrix from Ego-vehicle co-ordinate system to 
initial co-ordinate system. 


roto-translation matrix from sensor co-ordinate system to 
Ego-vehicle co-ordinate system. 


roto-translation matrix from object co-ordinate system to initial 
co-ordinate system. 


time shift vector. 


longitudinal contact point vector from Ego-vehicle dynamics 
model to road model. 


lateral contact point vector from Ego-vehicle dynamics model to 
road model. 


vertical contact point vector from road model to Ego-vehicle 
dynamics model. 


friction coefficient of road surface in contact with the Ego-vehicle 
wheels from road model to Ego-vehicle model. 

longitudinal gradient of road surface in contact with the 
Ego-vehicle wheels from road model to Ego-vehicle model. 


lateral gradient vector of road surface in contact with the 
Ego-vehicle wheels from road model to Ego-vehicle model. 


Framework Implementation 


logical scenario for driving situations with a stationary object on 
the side. 


logical scenario for driving situations with the Ego-vehicle 
driving around a stationary object. 


functional scenario catalog. 
set of logical scenarios with (n) dimension. 


9.5 List of Notations and Co-ordinate Systems 


9.5 List of Notations and Co-ordinate Systems 


Preliminaries 


AD measuring principle block that generates a detection list for each 
environmental perception sensor. 

B® processing block that generates a feature list for each 
environmental perception sensor. 

(et observation block that generates an object list for each 
environmental perception sensor. 

CI context element to define the safety requirements according to the 
various automated driving levels. 

C4 context element to use the test platforms according to the 
effectiveness and efficiency criteria. 

C3 context element to use a measurement system for field-based 
observations. 

C6 context element to define the criticality matrices for safety 
envelopes. 

El sub-evidence to define the coverage of functional requirements. 

E2 sub-evidence to define the coverage of value and time tolerances 
within software structures. 

E3 sub-evidence to define the coverage of system integration and 
variation. 

E4 sub-evidence to define the coverage of system performance. 

ES sub-evidence to define the coverage of training data and sensor 
uncertainties. 

E6 sub-evidence to define the coverage of driving scenarios. 

Gl main goal to provide a safety argumentation based on an accepted 
residual risk of an ADF. 

G2 sub-goal to track functional correctness for an ADF. 

G3 sub-goal to ensure the back-to-back consistency for a 


model-based code generator employed to software development. 


G4 sub-goal to specify the system integration and variation for 
deployment of an automated driving function. 


G5 sub-goal to present the software robustness. 
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G6 sub-goal to deliver the sensor availability and functional 
effectiveness of an automated driving function. 

G7 sub-goal to supply the software reliability and functional safety of 
an automated driving function. 

SI strategy of the GSN safety case based on the test termination 
criteria. 


Framework Conception 


Re camera reference co-ordinate system. 

Ro detected object point co-ordinate system. 

Re subject vehicle reference co-ordinate system. 
Rr inertial (global) reference co-ordinate system. 
RM 2D monitor reference co-ordinate system. 

Rs sensor reference co-ordinate system. 

Ro target object reference co-ordinate system. 
RR relative start reference co-ordinate system. 
RT track co-ordinate system. 


Framework Implementation 


P1 baseline response of the logistic regression function. 
P2 maximum response of the logistic regression function. 
P3 turning point giving a halfway response between the baseline and 


maximum of the logistic regression function. 
P4 curve slope of the logistic regression function. 
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